Evals on Aguilera Engineeringhttps://aguilera.ee/blog/tags/evals/Recent content in Evals on Aguilera EngineeringHugo -- gohugo.ioen-USeduardo@aguilera.ee (Eduardo Aguilera)eduardo@aguilera.ee (Eduardo Aguilera)Eduardo AguileraWed, 20 May 2026 00:00:00 +0000Evals before claimshttps://aguilera.ee/blog/evals-before-claims/Wed, 20 May 2026 00:00:00 +0000eduardo@aguilera.ee (Eduardo Aguilera)https://aguilera.ee/blog/evals-before-claims/<p>Most AI features ship on vibes. Someone tries three prompts, the output looks good, and it goes to production. Then it fails quietly on the fourth case and nobody notices until a customer does.</p> <p>Write the eval first. Pin the cases you care about. Measure before you claim the feature works.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">1</span><span class="cl"><span class="k">def</span> <span class="nf">test_extraction</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">cases</span><span class="p">):</span> </span></span><span class="line"><span class="ln">2</span><span class="cl"> <span class="n">passed</span> <span class="o">=</span> <span class="mi">0</span> </span></span><span class="line"><span class="ln">3</span><span class="cl"> <span class="k">for</span> <span class="k">case</span> <span class="ow">in</span> <span class="n">cases</span><span class="p">:</span> </span></span><span class="line"><span class="ln">4</span><span class="cl"> <span class="n">result</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">extract</span><span class="p">(</span><span class="k">case</span><span class="o">.</span><span class="n">input</span><span class="p">)</span> </span></span><span class="line"><span class="ln">5</span><span class="cl"> <span class="n">passed</span> <span class="o">+=</span> <span class="n">result</span> <span class="o">==</span> <span class="k">case</span><span class="o">.</span><span class="n">expected</span> </span></span><span class="line"><span class="ln">6</span><span class="cl"> <span class="k">return</span> <span class="n">passed</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">cases</span><span class="p">)</span></span></span></code></pre></div><p>An eval is not a unit test. It is a measurement you keep running as the model, the prompt, and the data drift underneath you. Treat the score as a contract.</p>Most AI features ship on vibes. Someone tries three prompts, the output looks good, and it goes to production. Then it fails quietly on the fourth case and nobody notices until a customer does.

Write the eval first. Pin the cases you care about. Measure before you claim the feature works.

1def test_extraction(model, cases):
2    passed = 0
3    for case in cases:
4        result = model.extract(case.input)
5        passed += result == case.expected
6    return passed / len(cases)

An eval is not a unit test. It is a measurement you keep running as the model, the prompt, and the data drift underneath you. Treat the score as a contract.

The discipline is old. The subject is new. That’s the whole job.

]]>