Skip to main content
Approach

How a morning report checks itself

aiosprocessagents

The morning stack is the run Data & Business Solutions starts the day with: a briefing, the meetings, the mail, and a check that the machines the work depends on are up. It exists because the shape of the morning was being re-decided every day, and a morning note that cannot be trusted is worse than no note. This is how the stack got a fixed shape, a second version that checks its own sources, and a list of what it still gets wrong.

Principal Consultant

The consultant's decisions on this project are mostly about trust.

On April 20 an audit of the morning run found about 20 failures across 5 phases. The consultant started a second version the same day and kept the first running, because a morning already depended on it.

On September 13 three of the morning skills ran on a cheaper model. Each ran inside its own copy of the session, so the cheaper model stayed inside that skill. It lasted one live run. The pilot ended the next morning with a rule that still stands: no cheaper model unless it is shown to match the one it replaces. Output that has to be reviewed and corrected costs time, and time costs more than tokens.

On October 9 the consultant built the morning by hand, ran it again through every skill, and compared the two. The skills won on the standup and the health check. The hand won on the mail.

Business Analyst

In the week of January 19 the analyst fixed the briefing's section order: weather, summary, calendar, chat, mail, tasks, actions. Action items became checkboxes, and a validation step checks the order before anything is saved.

In the week of February 2 the briefing was still skipping things, so the analyst wrote down the five failures that kept appearing, with what each one looks like, and put the table at the top of the skill. That table did not stop the easy path. It made the easy path easier to see.

From the week of February 9 the briefing reads the previous night's recap first, so anything left open shows up in the morning as a follow-up. If there is no recap, it says so and moves on.

The April 20 audit's 20 failures came down to 4 causes. A phase did the work inline instead of calling the skill that already does it. A task link was invented from a placeholder, and only one of those links was real. Sources were skipped, including a certificate check that was only a search of the inbox. And the run had no guardrails of its own.

PMO

On February 13 the separate skills became one run, with a session save between every step, because the run is long enough to compact somewhere in the middle and it should pick up from the last save instead of starting over.

The second version's center is one loop every phase ends with. Check that what the phase was supposed to produce exists and is shaped right. If not, fetch again without the cache and check again. After three tries, say DEGRADED and keep going. Do not pretend. Between April 20 and April 28 the second version wrote five verify reports, and all five say DEGRADED overall. On April 28 the briefing itself came back OK. The standup and the health check did not. The loop said so.

When chat and mail calls hung for 30 minutes twice in 8 days in July, the PMO planned the fix on July 28: move those reads to the command-line tool, which has its own timeout. It shipped on August 16, with the write side too, and the commit removes the last hard dependency on the call that hangs.

Developer

In the week of January 12 every briefing skill got the same gate at the top: the skill is not complete until its data is saved, because unsaved data is lost when a long session compacts.

In the week of February 16 mail, calendar, chat and notes each got a short command-line tool that returns a summary instead of the raw response, and on February 20 nine skills moved over to them. Mail also learned whose move a thread is, so a thread the consultant is waiting on does not look like one that was ignored.

On April 22 the developer added a hook that remembers every ID a tool actually returned in the session and warns when a note being written contains one it never saw. It warns. It does not block. The verify loop is what blocks.

Code Reviewer

The reviewer's work on this project was the April 20 audit and what it exposed two days later.

The audit's fourth cause, no guardrails, is the one that produced the second version. The verify loop caught its own contract the same afternoon: the audit step told the model to look a task up with a flag the task tool does not have, the live run failed on it, and the fix was to check every link against the task data already fetched in that session.

On April 22 the meeting prep invented four calendar events and attached real meeting quotes to them. That was not a missing check. A newer model was making things up more often, and other people were reporting the same. The fabrication guard hook is the answer the reviewer asked for.

QA

QA's record on this project is what the comparisons showed.

The September 13 pilot first. The cheaper model skipped the certificate check, miscounted errors, called untracked items tracked, and pulled a stale number from a week before. It wrote 59 lines against 112 on the session model. Two fixes the pilot found were kept: the health check now says how to find today's briefing, and the reading skill reads from disk instead of a draft that only existed in the calling conversation.

The October 9 side by side: the hand-built page reported one failed mail. The real number was 12. It also listed three code reviews waiting, and two of them already had a closed response. On the other side, the triage tool read 105 of 249 messages and found one action item, an advertisement, and missed all 5 real ones. The mail tool reads the newest 50 messages per account and has no way to ask for more. The work-item query never sent the version the server asks for, so every call came back bad, and a second project tree was invisible.

The verify gate passed both versions. It cannot tell them apart. A check that something exists is not a check that it is right.

Still open

Two tools return a wrong answer without failing, and a verify gate that only checks that a file exists will pass them. The skill text was fixed on October 9: triage results must match a count against a full pull, the cap that cut 10 real action items is gone, and a day with no meetings says so instead of writing nothing. The two tools themselves were not changed in their repositories as of that date. The next step is a gate that checks a count against the source.

Built Something with AI?

I'll spend 30 minutes telling you what my production-readiness review would flag.