Lab 1 of 6
141 labelled questions turn "feels better" into a number you can gate on.
A retrieval change that "looks better" in three hand-picked queries is a guess. A golden set is a fixed list of questions with the document that should come back. Every change is scored on the same list, so a regression has a number attached.
hit@5: for what share of questions is the right document anywhere in the top 5. MRR: the average of 1 divided by the rank of the right document, so rank 1 scores 1.0 and rank 2 scores 0.5. hit@5 says "found"; MRR says "found near the top".
make evalmode=hybrid k=5 n=141 hit@5=0.823 mrr=0.630 p50=7.9msMeasured on this repo, written to RESULTS-rag.json. The output line above is formatted from that file, not retyped. Needs the repo's Python venv; I have not run it from this page.
A small neural model gets train accuracy 1.0 on its own data. What is a fair guess for accuracy on held-out data?
Measured in RESULTS-p1.json: train accuracy 1, test accuracy 0.233 on a 98,823-parameter model. It memorised. Without a held-out set you would have shipped it.
1. Change k from 5 to 3 and predict which direction hit@k moves before you run it. 2. Find one question in the golden set where the right document is not in the top 5 and say why.