t1-p2-06authored3–5h

Re-run and paired before/after comparison

requires: t1-p2-05

Spec

Re-run the same 50-question eval set against the fix and report a before/after table broken down by category, not just an aggregate score. Commit evals/report.md, which must reference two run ids over the same eval set version.

Why this matters

The same eval set, or the comparison is meaningless. Reporting by category catches the common and demoralising outcome where the aggregate improves while your target bucket got worse and an unrelated one carried the score.

Verify

No result yet. Run `learn verify t1-p2-06` locally, or push to trigger CI.

artifactartifact: evals/report.md

unrun

reporepo: file_exists

unrun

judgejudge: evals/report.md

unrun

Rubric

# t1-p2-06 — Re-run and paired before/after comparison

**Task:** Re-run the same 50-question eval set against the fix and report a before/after table broken down by category, not just an aggregate score. Commit evals/report.md, which must reference two run ids over the same eval set version.

**Strong:** Directly answers every one of the review questions below in the affirmative, with specific evidence from the actual work, not a restatement of the task.

- Is the comparison genuinely paired?
- Are regressions reported, not just wins?

**Passing but weak:** The deliverable exists and technically satisfies the spec, but is generic, restates the task instructions instead of engaging with them, or only partially addresses the review questions below.

**Fails:** The deliverable is missing, or engages with none of the review questions below - e.g. it describes what was built without explaining the reasoning the task actually asks for.