Applied: evals, error analysis, and measurable improvement
Stop guessing whether changes help. Build the measurement apparatus first, then improve something and prove the improvement.
Definition of done
A committed eval report showing baseline vs improved scores, broken down by failure category, with the specific fix documented for each.
Tasks
t1-p2-01
authored
Hand-written eval dataset of 50 questions
5–9ht1-p2-02
draft
Automated eval runner
4–6ht1-p2-03
draft
Two-method grading layer
5–8ht1-p2-04
authored
Failure taxonomy from the first run
4–6ht1-p2-05
authored
One targeted fix for the largest failure bucket
4–7ht1-p2-06
authored
Re-run and paired before/after comparison
3–5ht1-p2-07
draft