t1-p2-01authored5–9h

Hand-written eval dataset of 50 questions

Write 50 question/expected-answer pairs by hand covering factual lookup (20), multi-document synthesis (15), and should-refuse / insufficient-info cases (15). Store as a versioned file at evals/eval_set.json.

By hand, deliberately. Generating an eval set with an LLM bakes the model's blind spots into the measurement, so the eval agrees with the system precisely where both are wrong. The 15 refusal cases matter most: without them nothing measures the "knows when it doesn't know" promise, and a system that confidently answers everything will score well.

No result yet. Run `learn verify t1-p2-01` locally, or push to trigger CI.

artifactartifact: evals/eval_set.json
unrun
reflectreflect
unrun