There's no test suite for a construction drawing

This month Anthropic wrote about how it keeps up with code written by AI. As of September it merged over 30,000 changes a day, up from about 2,500 in May. It built its own version control system to cope. Its Claude Code team says Claude now writes about 90% of that product's code.

Addy Osmani made the same point from another side. If you want an agent to do good work, give it a way to check its own work. If you can only have one kind of test, make it one that runs the whole thing end to end.

The debate has settled on one idea. Writing code is cheap now. Checking it is the job.

The check they assume

All of that advice rests on one thing. A cheap, automatic check already exists. Code compiles or it doesn't. Tests pass or they fail. An agent can run them a thousand times a day for almost nothing.

Most of my work has no such check.

What messy input looks like

On a client project for a US construction company, the input is a bid package. Drawings, spec books, spreadsheets, emails and zip files. Some drawings are clean. Some were marked up and re-saved so many times that the lines are mostly noise.

The AI reads all of it and lists the work the contractor has to price. Nothing tells you if that list is right. Only an estimator who has done it for years can. By hand, one bid takes them two to four days.

Sarvik had the same problem. Petitions came in as voice notes, photos of letters and typed messages, in five languages. No compiler tells you a Tamil petition went to the right government scheme.

So the check comes first

On messy input, the test suite is a set of real examples with the right answer written next to each one. Someone who knows the domain has to write those answers. It is slow work and nobody enjoys it.

It is also the only way to know whether a change made the AI better or just different. Without it, every new prompt is a guess that looks like progress.

Shrink what the model decides

Sarvik taught us a second rule. The model reads and drafts. It never makes up a number. Every figure on the screen came from government data, and every decision stayed with the MP.

When you can't test an answer, make the model responsible for less of it.

The part the debate skips

The labs are solving review for code, and code comes with a built-in check. Most useful work doesn't. Invoices, medical forms, legal letters, drawings, petitions.

If you're building AI for that kind of input, start by writing down what correct looks like. The model comes second.