Evals Are Not Tests

We’ve spent the better part of a year building an LLM layer over an analytics database, and the work doesn’t feel like writing tests.

A test wants a stable input and a stable answer. The function returns the value you expected or it doesn’t, and when it fails you usually know what broke. An analytics question is messier than that. Someone asks for revenue trends, but they might mean revenue for their segment, their date range, their team’s definition of revenue, the version of the metric that changed after the last board deck. They may not know how to say any of that clearly, and the model still has to turn whatever they typed into something the database can answer.

That’s why evals matter. But treating them like ordinary tests is the first place teams get them wrong.

The user’s question is never the real question

When someone types “show me revenue trends,” they don’t really mean revenue trends. They mean revenue trends for their segment, their date range, adjusted for the metric definition their team agreed on six months ago in a doc nobody reads. They want the number their VP brought up in the all-hands. They want something they can’t quite articulate, because they haven’t articulated it to themselves yet.

That’s not a failure of the user. It’s the nature of exploratory data access. Nobody knows exactly what they want until they see what they don’t want.

And that ambiguity doesn’t disappear when you put an LLM in front of a database. It moves. The user’s uncertainty becomes the model’s problem, except now the model has no way to ask the follow-up question the user could actually answer.

The model does not know it is wrong

LLMs are good at generating plausible SQL. They’re much worse at knowing when that SQL is wrong. A model will join the right tables, filter on the right column, group by the right dimension, and still hand back a number that’s off by thirty percent, because the real metric definition lives in institutional knowledge nobody wrote down. It doesn’t hesitate. It just returns the query.

The user, who isn’t a SQL expert, has no way to tell “correct” apart from “plausible-looking but wrong,” so they take the number to a meeting. This is where evals start to pull away from tests.

Evals measure alignment, not accuracy

A unit test asks a simple question: does this function return the right value? The ground truth is fixed, because you wrote it.

An eval for an LLM analytics system asks something fuzzier: did the user get what they needed? That’s harder, because the user isn’t always sure, because what they needed shifts as they explore, and because “correct” is sometimes close enough given the business context while “wrong” is sometimes just correct-but-off-by-one-key-dimension.

So the thing you’re measuring isn’t the model’s accuracy against a fixed answer key. It’s alignment: the fit between what a user can articulate, what the model interprets, and what the data actually says. Think of it as a signal, not a score.

The ratio matters more than the method

There are really two ways to measure that signal. Call them pooling and planning. Most teams lean hard on one or the other, and the ratio between them matters more than either method on its own.

Pooling is the reactive one. You collect real queries from production: what users asked, what the model generated, whether they engaged with the result or rephrased it straight away. Over time you build up a corpus of actual usage, failures included. It’s the most honest signal you can get, because it’s real people asking things in ways you’d never have thought to write down. The catch is that you’re always measuring the past. You find out about degradation after it has already happened. And if users quietly give up on certain questions because the answers kept coming back wrong, your pool grows blind spots. You never see the questions you trained people to stop asking.

Planning is the proactive one. Before you ship, you decide what good looks like. You write representative questions with known-good answers, set a baseline, and pick which failure modes actually matter: wrong SQL, or right SQL that answers the wrong question. The catch here is the mirror image of the other one. The questions you write at your desk aren’t the questions users actually ask, so you end up maintaining a synthetic eval set that drifts away from production faster than you can keep it current.

In practice I’ve drifted toward roughly seventy percent pooled, thirty percent planned. Not because I have a principled reason for that exact split, but because the synthetic set kept missing the failures that turned out to matter.

The planned set gives you a floor. It catches regressions on the cases you already know about, and it runs before every deployment. The pooled set tracks the drift. It tells you when the shape of the questions is changing, when people are rephrasing more often, which is really a proxy for the model no longer answering the question they meant.

That rephrasing rate is the signal I watch most closely. Someone asks a question, gets a result, then asks a slight variation of the same thing: that’s a failure. On its own it’s noise, but aggregated across hundreds of sessions it’s more honest than any eval set I could design by hand.

The feedback has to follow the version

The harder part is keeping that feedback meaningful across changes.

Model versions change. Prompts change. The schema changes. If you’re not tying user feedback to the exact model version and prompt that produced it, you can’t attribute anything. You’ll see that something got worse. You just won’t see what made it worse.

So I log three things for every query: the model version, a hash of the prompt, and whether the user engaged with the result or went back and modified it. That little triple lets me ask, after any change, whether the update helped this cluster of questions or quietly regressed it.

It’s not perfect. Users don’t always give clean feedback, and engagement is a proxy, not ground truth. But it’s about as close to ground truth as you get when the user themselves isn’t sure what correct looks like.

Tooling automates version tracking, not judgment

Doing all that logging by hand, across model versions and prompt iterations, gets old fast. And without somewhere to compare runs side by side, the logs are just noise anyway.

Braintrust is what fills that gap for me. The idea is simple enough: wrap your LLM call in an eval, define your test cases, and every run gets saved as a named experiment you can compare against the ones before it.

A basic eval looks like this:

from braintrust import Eval
from autoevals import Factuality

Eval(
    "analytics-queries",
    data=[
        {"input": "show me revenue trends", "expected": "revenue by month for the current fiscal year"},
    ],
    task=lambda input: ask_your_llm(input),
    scores=[Factuality],
)

You give it a question, a reference answer, and a scorer. The scorer here, Factuality, asks another LLM whether the output is grounded in what you expected. It’s not a perfect judge, but it’s consistent, and consistency is what catches regressions. You don’t need a perfect score. You need to know when the score drops.

Run it before every deploy. When you change a prompt or switch models, you get a diff: which cases improved, which regressed, and by how much. That comparison is what turns raw logging into signal. The part that matters most isn’t the score on any single run, it’s the trend across runs. Braintrust keeps every experiment, so you can look back and pin down exactly when quality started slipping and which change caused it. That’s the practical version of what evals are supposed to do: catch regressions before they reach production, instead of after a user files a ticket about a wrong number.

The frame is wrong from the start

I said at the start that evals sound like tests. But tests are for code, and code doesn’t change its mind. What you’re evaluating here isn’t code at all. It’s a moving target: the gap between what a user can ask, what a model interprets, and what a database can answer.

That gap never really closes. It just shifts. And the moment you start treating your evals as a one-time benchmark you passed, you stop watching the parts that are quietly drifting toward wrong.