Why a survival benchmark needs a fixture world
The scores got worse every run. The obvious reading was that the agent was getting less reliable. The real explanation was simpler and more embarrassing: the test arena was running out of trees.
A gather test sounds easy to score. Ask the companion to collect wood, count the logs it comes back with, repeat. Run it three times and average the result. The first version of the test did exactly that, in the same patch of world, run after run, and the numbers fell every time.
When a score falls across repeated runs, the instinct is to suspect the thing being measured: a bug that only shows up on the third attempt, state leaking between runs, fatigue in whatever passes for fatigue in a bot. None of that was it. The agent was chopping down the same forest every time to pass the test, and the forest does not grow back between runs. A WorldEdit count of oak logs in range fell from 110 to 50 over eight runs. The skill was fine. Testing it was using up the thing it depended on.
It is an easy mistake to make and an easy one to miss, because a falling trend looks exactly like a reliability problem until you go and check the world itself. A benchmark that changes the environment it tests in cannot tell "the agent got worse" from "the agent used up what it needed". Both draw the same falling line on the same chart.
The fix: a fixture world, restored before every run
The arena a survival test runs in is now a deliberate fixture. It is built once, checked into version control, and restored to its starting state before each run, instead of being left however the previous run left it. A gather test in run three faces exactly the same trees as the one in run one, so whatever the score says, it says about the agent.
26 of 27 task runs on Capabilities passed in an arena that starts from the same state every time.
This sounds obvious written down, and it is, once you have watched a capability metric quietly turn into a deforestation metric without anyone deciding it should. It is also why the numbers on that page can be trusted: nothing about the world changes between one attempt and the next.