Skip to main content
Build Blog

Build log: the test hub

testingagents

The hub is one Playwright project for the applications I maintain. I built it because each app was growing its own tests, the selectors were guesses, and a green run was not telling me the test had proved anything.

The week of March 16

I already had a scaffold of about 31 tests whose selectors were not mapped to a live page, so I made it one hub, with a shared sign-in, and I put it in its own repository. A review of that first commit found issues. I fixed the high and medium ones the same day. The follow-up review came back with none.

The tests had not been run against a live application.

The week of March 30

By April 2 the hub held 100 tests across 10 projects, and still none of them had been run against a live application. The selectors had been taken from reading the source. A challenge wrote that down. A test that has never touched the app is a plan, not a result.

The same days I designed a scanner and a report. The scanner maps pages before anyone writes a test, and the first design only follows links. It does not press a button. The report is one HTML file. A skill can draft a test from that map, and I accept or reject the file before it stays.

On April 3 I added a mode that does click. It opens tabs, search forms, and rows, and it refuses a button that would change or delete data. The first scan of that mode mapped 81 pages. The branch that day held 307 tests in 77 files. I opened the pull request. I had not watched those new tests run.

The week of September 21

The tracker still said 100 tests and 10 projects. The hub was at 587 tests and 11 projects. A task list is the wrong place for a run. It has no pass, no fail, and nowhere to put the screenshot.

I looked at a hosted tool that already stores cases. It reports to engineers. What I wanted was proof someone who does not write the tests can open, which is why I started the proof dashboard. It reads a hub run and says what the run proved. A run that goes quiet and never finishes is stopped. It is not passed.

The week of September 28

The dashboard reported 76 tests in one suite that could come back green without proving anything. 30 were scaffolds, a list of comments where the test should be. 34 ran and asserted nothing. 12 had every assertion inside a condition, so a false condition passed by skipping the check. That was out of 196 declarations in the source.

On October 2 I had the count re-derived, and then I read the tests. Some of the 76 were the checker being blunt, because an assertion lived in a helper the checker did not count. Some were real. I approved a fix in stages. Teach the checker about helpers. Rewrite the weak tests so a skipped condition is a failure. Delete a shell only after a real test already proves the same steps.

The same check showed saved sessions for a different host than the one the run was aimed at, and apps that were timing out or answering with a redirect. A session file can look fine because the cookie names are right. The preflight has to say which host those cookies belong to.

The week of October 12

A session on the Spark can build something and has no account on the machine that runs the hub. I want that session to ask for an end-to-end test. It hands over the address, the code, and the cases it already has. The hub writes what is missing, runs it, and writes a report back.

The first sketch was a shared folder, and I did not keep it, because the proof dashboard is the front door and the record and the hub is the engine. The dashboard does not run the tests. I wrote the hub side of that request the same week. The front door was the other half, and it was not done in that session.

The week of October 19

A PDF export had 3 paths that had only been checked by hand. The file itself, 2 prints at the same moment, and a print dropped in the middle. I said it should be tested from the hub. The plan starts the app on its own, makes a run to print, and checks the file is a whole PDF. I approved that plan the same day. The notes do not say the suite had run.

Where it is

The hub is one project, the scanner maps a page before a test is written, and the dashboard is where I look to see what a run proved. A passing test that never asserted anything is the bug I am still closing. A request from another session has a hub side, and the front door was not finished in the notes I have.

Built Something with AI?

I'll spend 30 minutes telling you what my production-readiness review would flag.