Blog ·
Why a survival benchmark needs a fixture world
The scores were getting worse every run. The obvious read was that the agent was getting less reliable. The real explanation was simpler and more embarrassing: the test arena was running out of trees.
A gather test sounds simple to score: ask the companion to collect wood, count how many logs it comes back with, repeat. Run it three times and average the result. The first version of that test did exactly this, in the same patch of world, run after run — and the numbers declined every time.
The instinct when a score declines across repeated runs is to suspect the thing being measured: a bug that only shows up on the third attempt, some kind of state leaking between runs, fatigue in whatever passes for fatigue in a bot. None of that was it. The forest the agent was chopping down to pass the test was the same forest every time, and it does not grow back between runs. A WorldEdit count of oak logs in range fell from 110 to 50 over eight runs — not because the skill was failing, but because the resource the skill depended on was being consumed by the very act of testing it.
This is an easy mistake to make and an easy one to miss, because a declining trend looks exactly like a reliability problem until you go and check the world state directly. A benchmark that changes the environment it is testing in cannot tell the difference between "the agent got worse" and "the agent used up what it needed" — both produce the same falling line on the same chart.
The fix: a fixture world, restored between runs
The arena a survival test runs in is now a deliberate fixture: built once, checked into version control, and restored to its starting state before each run rather than left as whatever the previous run turned it into. A gather test in run three faces exactly the same trees a gather test in run one did. Whatever the score says, it is a statement about the agent, not about how much of the world is left.
This sounds obvious written down, and it is — after you have watched a capability metric quietly become a resource-depletion metric without anyone deciding that was the plan. It is also why the numbers on Capabilities are worth trusting: 26 of 27 task runs passed in an arena that starts from the same state every single time, not one that degrades a little further with every attempt that came before it.