Jake Goldsborough

Evaluating Infrastructure Agents in Running Systems

Aug 02, 2026

5 min read

#ai

#rust

#infrastructure

#dev

I have been building Replaybook as an incident replay trainer.

It starts a broken service, gives you a workstation with Docker access, and asks you to fix the incident. The success condition is objective. Usually it is a health endpoint or a real connection to a dependency.

That was useful for humans. Then I wanted to know how agents would do.

Not on a coding benchmark. On a running system with containers, state, bad configuration, and a verifier that restarts the service after the agent says it is finished.

The stack

The evaluation harness is four projects working together:

Each trial gets a fresh NixOS VM, a fresh Docker environment, a separate SSH port, and only the credentials required by that agent.

The agent does not get a repository with a failing test. It gets an incident report:

The application is receiving PostgreSQL authentication failures.
The database container is running, but TCP connections are rejected.

Investigate the database container from the incident workstation and restore
a real TCP connection. Make the fix survive a restart.

It has to inspect the running containers and repair the deployed service.

Traditional coding benchmarks usually look like this:

Replaybook starts with a running system instead:

The difference is state. The thing being evaluated is not just an artifact in a checkout. It is a system that has already been started and can retain, or lose, changes over time.

The verifier defines the benchmark

A passing health check is not evidence of a repair.

An agent can launch a second nginx process on the expected port, get a 200 response, and report success. The verifier restarts the container, the original broken configuration comes back, and the benchmark correctly fails. Manually started processes disappear. Ad-hoc shell changes disappear. Only a modification to the deployed system survives the restart.

The verifier checks the user-facing path, restarts the affected services, then checks again:

if ! wait_for_health; then
  fail
fi

docker restart "$app_container"
docker restart "$nginx_container"

if ! wait_for_health; then
  fail
fi

Restarting the service measures durable repair instead of temporary success. That one restart changed the results.

The first matrix

I ran claux against three scenarios:

Three attempts per scenario. Nine trials total.

The result was 7/9, at a reported cost of six cents.

That is not a spectacular benchmark score. It is much more interesting than one.

The PostgreSQL scenario passed three out of three. The missing environment scenario passed two out of three. The Nginx scenario also passed two out of three.

One failure was a temporary fix that worked until the restart. Another agent timed out after fifteen minutes without recovering the application.

Those are different failures. The verifier tells me which one happened instead of reducing both to “the model got a zero.”

What this is measuring

The agent has to do more than write a plausible command.

It has to:

That is closer to incident response than code generation.

Traditional coding benchmarks ask whether an agent can produce the correct artifact. Replaybook asks whether the running system is actually healthy afterwards. The useful distinction is between making the current check green and changing the system into a healthy state that stays green.

The part I did not expect

I started this because I wanted to compare models. I assumed the interesting output would be a leaderboard.

The more useful output is the failure classification.

A failed trial can mean:

Those are all different engineering problems. A single reward number hides most of them.

What is next

I want to add more scenarios that test different operational skills:

I also want to run the same matrix with Codex and Claude Code once the authentication setup is less annoying.

This is still a small personal tool. It is not a serious benchmark yet.

But the evaluation harness is real now. Same incident, fresh machine, objective verifier, repeated trials, and enough detail to understand why an agent failed.

That feels more useful than asking whether a model is “good at coding.” Understanding why an agent failed is more valuable than a single benchmark score.

Related Posts