Stop evaluating the AI model. Evaluate the system

Every time a new AI model is released, benchmark scores go up, and we treat that as progress. But most teams are not shipping standalone models. They are shipping systems: agents with tools, retrieval, memory, and escalation paths.

At CyberCare AI Labs, our AI Developer Rokas works on customer-support AI agents that need to function inside real products, not just perform well in demos. Here are three things that changed how we evaluate AI agents in production.

1. The evaluation is the spec, not the test

In normal software, you write requirements first, then write tests against them. The requirement is the source of truth.

That breaks down with LLMs. “The assistant should respond helpfully” is not something an engineer can build against. It only becomes real when you write the evaluation that defines what “helpful” means: which inputs matter, what good outputs look like, and what counts as a pass.

So the evaluation becomes the operational spec. It is where “works” is defined precisely enough to act on. Change the evaluation, and you have changed the product.

That is why AI agent eval is not something you write after the prompt looks good. It is something you write first, and the prompt becomes one implementation of it.

Once you work this way, prompt iteration stops being a taste debate. Two prompts, two models, two architectures – the evaluation gives you a number. Without one, every change becomes an argument about whose intuition is sharper.

2. You can’t buy your eval

When teams start thinking about AI agent evaluations, there is a strong temptation to adopt off-the-shelf standards or mimic frontier lab leaderboards. Someone must have already built an evaluation. Surely we can reuse it.

You can. But it will tell you very little.

Public AI benchmarks measure general model capability. Your agent is not doing general work. It is doing your job: in your product, on your data, for your users, with your tools.

The inputs it sees do not look like benchmark inputs. The failures that matter to you are not the failures the benchmark was designed to catch. The more specialized your AI agent is, the weaker the signal a public benchmark score carries.

This brings us back to point one. If the evaluation is your spec, and your spec is unique to your product, your evaluation cannot simply be borrowed. It has to be built around the reality of your system:

a dataset of real examples from your traffic,
a rubric written for the behaviors that actually matter to your users,
a grading process you calibrate yourself.

It is more work than installing a library, and it is the only way to know whether what you are shipping is actually good.

3. The right answer is only a starting point. Measure the path

When teams first build agent evals, they usually start by grading the final output. Correct answer? Pass. Wrong answer? Fail. That is the right place to start. It is cheap, it is fast, and it answers the first question worth asking: is the end result good enough at all? If outcomes are failing, you do not need trajectory data to know you have work to do.

The mistake is staying there too long.

Because an agent’s output is the end of a process: tool calls, retrievals, intermediate decisions, recoveries from its own mistakes. Once your outcomes look good on average, outcome grading goes quiet on the things that bite you later. A correct answer reached the wrong way is a bug waiting for scale. Usually it surfaces in front of a customer.

That is when you drop a level. Trajectory evaluation asks a different question. Not whether the agent got the right answer, but:

• Did it get there the right way?
• Did the agent call the tool you expected, with arguments that make sense?
• Did it recover when a step failed, or did it stumble onto the answer by accident?

Outcomes tell you whether the system works on average. Trajectories tell you whether it works for the reasons you think. That distinction is where the agents that look great in a demo come apart in production: the one that answered correctly after four redundant API calls, the one that got it right because the user’s question happened to be easy. You won’t catch those without grading the path.

Engineering, not guessing

At CyberCare AI Labs, we ship customer-support AI agents into real products. The bot handles what it can; a human agent picks up what the bot cannot.

Evaluation is what makes that handoff trustworthy. Without it, we would be guessing whether a prompt change made things better, whether a new model is worth the migration, or whether the retrieval rewrite actually improved anything.

The leaderboard tells you the AI model is smart. The evaluation tells you the AI agent system works.