AI Evaluation at a Production Level

The demo went well. The model answered the questions, the room was impressed, and the project got its green light. A few months later the same system is handling real customers. Is it still doing a good job? In most teams we've seen, nobody knows. The demo proved the system could work once. Nobody set up a way to know whether it works now.

So how do you evaluate an AI system so that "it's working" is something you can show instead of something you feel?

Vibes are not evaluation

The default way teams check an AI feature is to try some inputs and read the outputs. If the answers look reasonable, it ships. We get why. It's fast, it feels concrete, and the results are usually encouraging, because people naturally test the cases they already expect to work.

The problem is that a dozen good answers only prove the system can be right. They don't tell you how often it's wrong, on what kinds of input, or how badly. And spot-checking breaks down exactly when it matters most. At ten users a day, you can read every output yourself. At ten thousand, the failure you never thought to test for is happening every day, to real people, and nobody is reading those outputs anymore.

Decide what "good" means before you measure it

Our method for every project, CRISP-DM, puts business understanding first, before the data and before any modeling. Evaluation is where that order pays off. You can't score a system until you've decided what a good outcome is, and that decision belongs to the business, not the model.

A model metric is not a business outcome. A system can score 95 percent on a benchmark and still fail you, if the 5 percent it misses are your most expensive cases, or if the benchmark looks nothing like a normal Tuesday. The useful questions sound like a manager asking them:

• What decision does this output feed, and what does a wrong output cost?
• Which mistakes are cheap and which are expensive? Missing a fraud case and flagging a good customer are both errors, but they don't cost the same.
• What's the honest bar for shipping? "Better than the manual process we run today" is often a more useful target than an accuracy number.

Write the answers down before you look at a single score.

Build a test set you can trust

Once you know what good means, you need a fixed set of examples to measure against. This is the backbone of everything else, and it's worth real effort.

A good test set looks like production. It's drawn from the same messy stream of real inputs, and it includes the hard cases on purpose: the ambiguous ones, the strange formats, the questions people ask that you wish they wouldn't. Each example needs a trusted answer attached, decided by someone who knows the domain. Scoring against wrong answers is worse than not scoring at all.

And once it's built, version it. Freeze the set, number it, and keep it stable, because the whole point is comparing results over time. When you add new cases, that's a new version, and you rerun the old system against it so the comparison stays fair.

Using models to judge models, carefully

Checking outputs by hand doesn't scale, and for open-ended work like summarizing there's often no single right answer to check against. A common fix is to use a strong model as the judge: it scores another model's output against a rubric. Done well, this lets you evaluate thousands of cases instead of the few dozen a person has time for.

Done carelessly, it produces confident numbers that mean very little. A model judge has its own habits. It can favor longer answers, reward confident wording over correct content, and share the blind spots of the model it's grading.

So calibrate it like any new instrument. Have the judge score a sample, have a person score the same sample, and compare. Where they disagree, fix the rubric, or accept that a human has to make that call. Keep people on the decisions that matter most.

Evaluation doesn't end at launch

The hardest habit to build is treating evaluation as ongoing instead of a gate you pass once. The world keeps moving after launch. Your users change. The model version changes. Someone tweaks a prompt to fix one thing and quietly shifts ten others. Any of these can degrade a system that passed every check on day one.

In practice, staying honest takes three habits:

1. Watch real production outputs, not just the offline scores, so you notice when the inputs drift away from what you tested.
2. Collect feedback from real users, and feed the cases where the system failed back into the test set.
3. Rerun the full test set whenever anything changes: the model, the prompt, the data. No exceptions, because the one time you skip it is the time something broke.

None of this is exotic. It's the same care good teams already put into the rest of their systems, applied to the part that can give different answers on different days.

If you have an AI system in production and can't confidently say how well it's doing, we're happy to talk through how we'd set up evaluation for your case.