t1-p2-04authored4–6h
Failure taxonomy from the first run
requires: t1-p2-03
Spec
Manually categorise every failure from your first eval run into buckets - retrieval missed relevant chunk, hallucinated fact not in context, wrong source cited, over-confident when it should have refused, verbose or unclear. Commit evals/failure_taxonomy.json and a readable report.
Why this matters
Manually, and every failure. Aggregate scores tell you the system is 68% good and nothing about what to do next. The bucket counts are what turn a number into a decision, and doing it by hand once is what teaches you to see the categories.
Verify
No result yet. Run `learn verify t1-p2-04` locally, or push to trigger CI.
artifact
unrun
judge
unrun
Rubric
# t1-p2-04 — Failure taxonomy from the first run
**Task:** Manually categorise every failure from your first eval run into buckets - retrieval missed relevant chunk, hallucinated fact not in context, wrong source cited, over-confident when it should have refused, verbose or unclear. Commit evals/failure_taxonomy.json and a readable report.
**Strong:** Directly answers every one of the review questions below in the affirmative, with specific evidence from the actual work, not a restatement of the task.
- Are categories mutually exclusive and specific enough to act on?
- Does any bucket amount to "miscellaneous"?
**Passing but weak:** The deliverable exists and technically satisfies the spec, but is generic, restates the task instructions instead of engaging with them, or only partially addresses the review questions below.
**Fails:** The deliverable is missing, or engages with none of the review questions below - e.g. it describes what was built without explaining the reasoning the task actually asks for.