Escalating Evals: 72% Fewer LLM Judge Calls

The cheapest LLM-as-judge is not doing the judge at all! While the Jev output is not on-par with the LLM-as-judge, using it with escalation gives us great cost saving with small amount of difference in verdicts

The cheapest LLM-as-judge is not doing the judge at all! While the Jev output is not on-par with the LLM-as-judge, using it with escalation gives us great cost saving with small amount of difference in verdicts

Escalating Evals: 72% Fewer LLM Judge Calls

Most of my evals end with the same expensive step: an LLM judge reads what the agent produced and rules pass or fail. When you’re running hundreds or thousands of acceptance checks, the cost starts to matter. I have spent time with Apo, the open-source agent-testing tool I build, making sure similar judge calls maximize the prompt-cache hits.

But there is an even cheaper way than cached calls:

Not making the call at all!

A new class of tiny decision models can help here. I use Jev from TypeSafe AI, which returns a verdict plus a calibrated confidence for roughly a hundredth of a cent per check. Across my 2,094 real checks, when it answers with confidence of 0.95 or higher, it agrees with my full LLM judge 99.5% of the time. In my Jev-as-judge blog I used it as a confidence signal besides the LLM-as-judge, but we can also use it as a replacement for the high confidence checks. This way we can escalate the harder checks to the LLM-as-judge and keep high confidence checks only on Jev.

And this blog is about that. How it helps? How is the output quality? How it compares to the old way of handling?

TL;DR Escalating Jev is a trade-off, we would have higher score by just having LLM-as-judge, but having Jev as initial check makes the tests cheaper without changing many verdicts compared with the LLM-as-judge.

Jev as escalation step

What I in my previous Jev blog showed, is that when Jev is very confident in its answer (>=0.95 confidence), it agrees with my LLM-as-judge result 99.5% of the time!

So the question is, why not just use Jev in these situations, why I’m paying for LLM-as-judge when the Jev is confident.

And that is exactly what my new Apo judge mode is about: Escalating to the LLM when it is needed, keeping Jev as initial scorer.

In my testing, 72% of all the checks were these confident answers, so we can make most of the checks that run shown with only the Jev check on the tests I previously did. This is massive savings on money, around 70% cheaper judging overall by removing the 72% of the checks with the Jev only result.

That 72% comes from my own suite, and the total judging bill for all 2,094 checks was $0.20. Your number depends on what your checks ask. Mine are mostly read-off checks, does the deliverable contain the right thing, which is exactly the kind a confident cheap model settles. Checks that ask for a derived judgment should always go to the full judge.

And that is the new Apo mode:

my-task.eval.ts
test("sla-credit-cap", async (t, { deliverables }) => {
  await t.judge(
    deliverables.redlinedDocument,
    "PASS when the SLA credit cap is marked up from 15% to 30%.",
    { judge: { mode: "cascade" } }, // this test only
  );
});
sla-review.eval.ts
task("sla-review", {
  judge: { mode: "cascade" }, // every t.judge in this task
  adapter,
  deliverables,
});

Escalation won’t improve the score

When we start comparing the score of escalation to only using LLM-as-judge, we know that the score won’t improve. And there is already research on this topic.

When the cheap judge is confidently wrong, the LLM-as-judge is also 96% of the time wrong on the same check. So it’s not like Jev itself is any better judge.

What the overall score is when cascading:

Acccascade=(1−ε)⋅AccJevconfident+ε⋅AccLLM-as-judge\mathrm{Acc}_{\text{cascade}} = (1 - \varepsilon) \cdot \mathrm{Acc}_{\text{Jev}}^{\text{confident}} + \varepsilon \cdot \mathrm{Acc}_{\text{LLM-as-judge}}

where ε is the escalation rate, so 1 − ε = 72% is the share Jev settles alone

The Jev part is incredibly cheap, so the more checks Jev settles on its own (the lower ε is), the cheaper the tests.

However, score is something like this:

AccJev only  <  Acccascade  ≈  AccLLM-as-judge\mathrm{Acc}_{\text{Jev only}} \;<\; \mathrm{Acc}_{\text{cascade}} \;\approx\; \mathrm{Acc}_{\text{LLM-as-judge}}

and

∣Acccascade−AccLLM-as-judge∣  ≤  (1−ε)⋅d  =  0.72×0.005  ≈  0.4%\left| \mathrm{Acc}_{\text{cascade}} - \mathrm{Acc}_{\text{LLM-as-judge}} \right| \;\leq\; (1 - \varepsilon) \cdot d \;=\; 0.72 \times 0.005 \;\approx\; 0.4\%

This calculation is actually interesting, it shows, from my own data, the upper bound on how many verdicts can differ from the LLM-only setup. Only 0.4%! What that means is that when Jev is confident, and the LLM verdicts differ, that is just 0.5% of the confident 72%, about 0.4% of all checks. Otherwise the output quality is the same as running the LLM-as-judge only!

I find this very convincing: roughly 70% cheaper judging, while at most 0.4% of the verdicts can differ from what running only the LLM-as-judge would have said. Which way those few verdicts go, better or worse, I can’t tell without labeled ground truth. The score could even go up, if Jev is right where the LLM judge was wrong. I think that’s worth the trade off.

The new mode is now in Apo! And I’m quite hyped about it, I can get the cost of the judging much lower, without losing much of the quality.

But it’s not by default. I still believe that running both can be helpful, even if the current data that shows it is convincing.

But all this has to do with the quality of the checks. If the LLM-as-judge checks are done well, the escalation rate will get higher and higher, and we can enjoy cheaper evals!

Back to Blog