Project verification harness

The cost of verification grows faster than software

The cost of constructing trustworthy verification infrastructure grows faster than the cost of constructing the software being verified.

For a tiny task, an agent can often build both the thing and a stronger harness around it:

task complexity:        1
verification harness:   3

That is fine. You spend more effort proving the thing than building it.

But then the software grows:

task complexity:        10
verification harness:   30

Still feasible.

Then:

project complexity:        1,000
bespoke verification:      10,000+

At that point people stop building the verifier. Not because verification has become less important, but because every project would need to reinvent a second, even more complicated software system whose job is to create environments, provision dependencies, generate states, orchestrate services, run browsers, inspect databases, inject failures, collect evidence, understand regressions, manage isolation, and report what actually happened.

So development keeps scaling while verification investment collapses relative to development complexity.

The resulting failure mode

That gives you the failure mode you’re pointing at:

small task
    ↓
build implementation
    +
build thorough verifier
    ↓
high confidence


large project
    ↓
build implementation
    +
"we should really build..."
    ↓
unit tests + mocks + a few E2Es
    ↓
unknown behavioural surface
    ↓
bugs

Amortise the verifier across every project

And yes: Testkit can be understood as amortising the cost of the verifier across every project.

Instead of this:

Project A → bespoke verification system A
Project B → bespoke verification system B
Project C → bespoke verification system C
Project D → bespoke verification system D

you want:

                    ┌─ Project A
                    ├─ Project B
Testkit verifier ───┼─ Project C
                    ├─ Project D
                    └─ ...

The application-specific part then becomes comparatively small: describe what must be true and provide whatever domain-specific actions/oracles Testkit cannot infer.

Testkit owns the expensive generic machinery.

One principle through a project’s lifetime

That means the same principle can hold continuously through a project’s lifetime.

The developer shouldn’t have to build a miniature distributed-systems testing platform to verify that migration. That platform should already exist.

A sufficiently powerful experimental apparatus

This also explains something important about Testkit’s scope. A conventional testing library attacks:

“How do I write this test?”

Testkit is attacking something closer to:

“How do I cheaply obtain a sufficiently powerful experimental apparatus for whatever software I’m currently building?”

That is a much bigger idea.

Agents make the asymmetry worse

And there is a particularly important agentic angle. Agents make the asymmetry worse.

An agent can produce 5,000 lines of implementation surprisingly cheaply. It cannot thereby make the verification problem 5,000 lines cheaper. If anything, cheap implementation means software complexity can accumulate faster than humans traditionally allowed it to.

So without something like Testkit:

cost of producing software ↓↓↓
cost of proving software     ~same

⇒ verification debt explodes

Whereas Testkit’s intended effect is:

cost of producing software ↓↓↓
cost of constructing serious verification ↓↓↓

⇒ confidence can scale with production

The central thesis

AI dramatically lowers the marginal cost of creating software, but not the marginal cost of creating bespoke infrastructure capable of thoroughly verifying that software. Testkit turns that verification infrastructure into shared reusable machinery, so rigorous behavioural verification can remain economically viable from a one-function task through to a large production system.

An empirical development loop

And there is a second-order consequence which I think is critical: if Testkit succeeds, an agent doesn’t just become better at testing after implementation. It can start choosing implementations based on whether they are experimentally verifiable.

That changes the development loop from:

agent writes code
      ↓
hope tests are adequate

into:

agent receives goal
      ↓
construct/verbalise observable success criteria
      ↓
Testkit provides experimental environment
      ↓
agent implements
      ↓
real system is exercised
      ↓
evidence contradicts or supports claim
      ↓
agent iterates

At that point Testkit starts looking less like a test runner and more like the empirical substrate for agentic software engineering.