humanjavalearn with RSS
Ep 4New

How do you know it's any good?

Evaluation

2026-11-21

Audio is still in production. Read along for now — the narrated version lands here soon.

About this episode

Evaluation is the least glamorous idea in applied AI and the one I most wanted to get right. It is the difference between "I built a thing that seems to work" and "I can say, with evidence, that this is safe to turn on." This one is me thinking out loud about offline evals, online evals, using a model to judge a model, and why an eval set is really just acceptance criteria you can run on every commit.

Two voices: a Host asking the questions, and me working through the answers.

Transcript

HOST: Today's topic is evaluation. And I'll be honest, that sounds like the vegetables of AI engineering. Everybody says you should do it, nobody wants to talk about it.

VERGEL: Fair. But here is the reframe, and it is the whole reason I wanted to start here. Evaluation is how you earn the right to ship. It is the difference between a thing that seems to work in a demo and a thing you can actually put in front of people and trust. One of those gets you applause. The other gets you deployed.

HOST: So why is it hard? If you build something, can't you just look at it and see if it is good?

VERGEL: You can, for about five examples. Then it falls apart, for a specific reason. With normal software, two plus two is always four. You write a test, it passes or it fails. With a language model there is no single right answer. Ask it the same question twice and you get two different wordings. "Looks good" does not scale, and your gut is not a measurement.

HOST: So what replaces the gut?

VERGEL: Two things, and you want both. The first is offline evaluation. You build a fixed set of inputs, each with something you expect to be true about the output, and you run that whole set every single time you change anything. It is a regression suite. The same idea as software tests, just adapted for outputs that vary.

HOST: Is that instinct new for you?

VERGEL: Not at all, and that is the fun part. When I rebuilt NostrKey into a product across four platforms in five days, the only reason I could move that fast was the test suite. Six hundred and thirty-plus automated tests meant every change was gated. I was never guessing whether I had broken something. Evaluation for AI is that same instinct, pointed at a system whose outputs are not deterministic. The suite is the thing that lets you move fast without moving blind.

HOST: Okay. But where do the test cases come from? That feels like the part where you could fool yourself, by only testing the stuff you already know works.

VERGEL: That is exactly the trap, and you just named the most important skill in the whole discipline. Building the eval set is where the quality actually comes from. You want cases that are representative of real traffic, and you want adversarial cases, the weird and the hostile inputs, drawn from what real users actually do. Not the happy path you are hoping for. The unglamorous truth is that curating that dataset matters more than which model you pick.

HOST: You said two things. Offline was the first. What is the second?

VERGEL: Online evaluation. Judging the system while it is live, on real interactions. Offline tells you "did this change make it better on my fixed set." Online tells you "is it actually good out in the world, on inputs I never thought of." You need both, because the world is more creative than your test set.

HOST: And how do you judge something live, at scale? You can't have a human read every one.

VERGEL: This is where it gets interesting. You can use a model to grade a model. You give a second model a clear rubric and ask it to score the output.

HOST: That sounds a little like asking the fox about the henhouse.

VERGEL: Your instinct is correct. It is called model-as-judge, and it has real failure modes. A judge can be biased. It can be gamed. It can favor the first answer it sees just because it is first. So you do not trust it blindly. You check the judge against human ratings on a sample, you give it a specific rubric instead of a vague "is this good," and you keep a human in the loop for the high-stakes calls. Used carefully, a judge lets you measure quality at a scale no human team could reach.

HOST: So what are you actually measuring? "Good" is still fuzzy.

VERGEL: Right, so you break it into things you can name. For a system that takes actions you care about task success, did it actually accomplish the goal. Faithfulness, did the answer stay true to the facts you gave it or did it drift. Answer relevance, did it address what was asked. And then the boring but decisive ones, latency and cost per task. "Good" is all of those, measured separately, so when something slips you know which dial moved.

HOST: And how do you even see all that? There are a lot of moving parts.

VERGEL: That is observability, or tracing. You log every step of a run so you can replay it and find where it went wrong. There are good tools for this now, Langfuse, Braintrust, Phoenix. The point is simple. If you cannot see inside a run, you cannot improve it. You are just shipping and praying.

HOST: This connects to something you have written about, right?

VERGEL: It does. I wrote a piece on keeping facts in retrieval and behaviour in the weights, about when to reach for retrieval versus fine-tuning. Evaluation is the thing underneath both of those. It is how you prove either one actually worked, instead of just believing it did. Whatever you build, the eval is what turns a belief into a fact.

HOST: Give me the one big idea to walk away with.

VERGEL: An eval set is acceptance criteria, made executable. I have spent twenty years taking a vague, ambiguous ask and turning it into a bounded problem with clear success criteria. For an AI product, those success criteria are the eval set. So the requirements stop being a document everyone argues about and become a thing that is either green or red, on every commit. That is the magic. The contract becomes something you can run.

HOST: Before we close, the test. What should someone be able to say out loud after this one?

VERGEL: Three things. One, the difference between offline and online evaluation, with an example of each. Two, three ways a model-as-judge can mislead you, and how you would guard against each. And three, the real one. Someone says "the system sometimes gives wrong answers," and they cannot tell you if it is the retrieval, the facts, the model, or the prompt. Can you walk them from that complaint to an eval set and a passing bar, out loud, in two minutes? If you can do that last one, you do not just understand evaluation. You can use it to earn trust.

HOST: Vegetables well and truly eaten.

VERGEL: Best thing on the plate. Next time we go back a step, to the thing that feeds all of this: how retrieval actually works. See you there.

Things I'm still chewing on

  • Three ways a model-as-judge can quietly mislead you, and the cheapest guard against each.
  • What actually belongs in a first eval set, if you only had time to write ten cases.
  • Where "acceptance criteria you can run" changes how you would scope a project with a client.