>
eval_set: refund_qa · 240 examples
metric: rubric · pass-rate
OPTIMIZING PROMPT...
. . ✶ . . . ✶ ✶ ✶ . . ✶ ✶ ✶ ✶ . . ✶ ✶ ✶ . . . ✶ . .
v3 -
v5 -
v7 -
Latest Writing
Thoughts on software development, AI, and building systems
Engineering Cheaper LLM Evals
Engineering solutions I have found when trying to implement a cheap way to do thousands of eval checks. From prompt structures and its cache points to the Jev and escalation modes for evals.
Jev vs OpenAI Decisions: 2,094 Real Eval Checks
Does it matter what System One model you should be using? On same tasks, how well do Jev and OpenAI's Decision API agree, and which one turns out to be stronger
Escalating Evals: 72% Fewer LLM Judge Calls
The cheapest LLM-as-judge is not doing the judge at all! While the Jev output is not on-par with the LLM-as-judge, using it with escalation gives us great cost saving with small amount of difference in verdicts