Lab 3 of 6
Drag the allowed drop. Watch which candidates would have merged.
A gate runs the golden set on a candidate and compares hit@5 with the current champion. If the drop is larger than the tolerance, it exits non-zero and the pipeline stops. Nobody has to remember to check.
Baseline is the hybrid retriever at hit@5 0.823. Each row is a candidate scored on the same set.
At a tolerance of 0.020, keyword-only and vector-only are blocked and the reranked candidate merges. Slide it to 0.100 and keyword-only slips through, which is the point: the threshold is a decision, and the gate makes it visible.
In this repo the real gate uses MAX_DROP = 0.03 (mlops_lab/gate.py). At 0.030 the vector-only candidate (drop 0.036) is still blocked.
make gate-drillThe target passes a good candidate and requires a bad one to be blocked (gate --bad must exit non-zero). Not run from this page.
Choose a tolerance for a product where a wrong answer costs money. Write one sentence for why it is not 0.000.