t1-p2-02draft4–6h
Automated eval runner
requires: t1-p2-01
Spec
Run all 50 questions through the pipeline, recording per question the retrieved chunks, generated answer, latency, and token cost. Write one run file per execution to evals/runs/<run-id>.json, referencing the eval set version used.
Why this matters
Recording retrieved chunks alongside the answer is what makes the failure taxonomy possible. Without it you can see that an answer was wrong but never whether retrieval or generation caused it - and those need opposite fixes.
Verify
No result yet. Run `learn verify t1-p2-02` locally, or push to trigger CI.
contract
unrun
artifact
unrun