A team building a customer-support agent did the responsible thing and built an evaluation harness before shipping. They needed a backing database for the agent to read and write during each test episode: customers, tickets, orders, message threads. So they hand-built a small one, wrote a few dozen scenarios against it, and watched the agent score in the low nineties. They shipped. The first week of real tickets was a different story, and every failure traced back not to the agent but to the toy database it had been evaluated against.
An agent evaluation is only as honest as its environment. If the world the agent is tested in cannot produce the situations the real world produces, the eval cannot catch the failures the real world will. Here is what the toy database could not do, and what each gap let through.
The tickets had no coherent state
In the hand-built data, a ticket's status was just a value someone typed. So the eval never contained a ticket that had legitimately moved open → pending → resolved → reopened, and it never contained the illegal states the agent needed to refuse to act on. Live, the agent cheerfully "resolved" a ticket that was already closed, because it had never been tested against a status it should have questioned. A real environment needs statuses that walk a legal state machine, with the histories that produce them.
ticket status event_ts TK-8841 open Mon 09:12 TK-8841 pending Mon 14:03 TK-8841 resolved Tue 10:20 TK-8841 reopened Thu 16:44 a real lifecycle the agent must reason about
The references did not resolve
Tickets pointed at customers, orders pointed at customers, messages pointed at tickets. In the toy data those links were approximate, so the eval never punished the agent for acting on a dangling reference. Live, it promised a refund on an order id that belonged to no one, because in testing every id it invented happened to work well enough. An environment an agent can be trusted against needs every foreign key to resolve, the same way a production system does.
Nothing connected across the episode
The hardest part of agent evaluation is multi-step coherence. The agent looks up a customer, reads their order history, checks a ticket, takes an action, and every one of those reads has to agree with the others or the episode is nonsense. Hand-built fixtures fall apart here fastest, because keeping a dozen tables mutually consistent by hand is exactly the work you cannot do at the scale an eval needs. The agent had been graded on episodes that never cohered, so it learned that coherence did not matter.
What changed when the environment got real
They rebuilt the eval environment with CrossRow, starting from the customer-support template and adapting it. What they got was a backing system that behaves like production: ticket statuses that walk a legal state machine, customers and orders and messages whose foreign keys all resolve, and histories that stay consistent across every table an episode touches. Crucially, it is generated with no real customer data, so the eval environment carries no privacy exposure even though it reads like the real thing.
The agent's score dropped. That was the point. The failures that had been invisible against the toy data were now reproducible in the harness: the reopened ticket it should have questioned, the dangling order it should have refused, the multi-step episode where its third action contradicted its first. Every one became a test they could fix against before the next customer saw it.
One honest boundary. CrossRow generates the environment the agent acts in, the coherent stateful database behind the tools, not the agent, its prompts, or its trajectory. But that environment is the part that is hardest to fake and the part a weak eval quietly skips. A high score against an incoherent world is not a passing grade. It is a measurement of the wrong thing.