← All projectsIntermediate · About 8–10 hours
Eval harness with a CI gate
Turn "it looks fine" into numbers that block a bad prompt change from shipping.
Good for: AI engineer, ML / MLOps engineer
What you will have at the endA test suite that scores an LLM feature on a golden set and fails the build when quality drops below a threshold.
- Golden datasets
- Assertions and LLM-as-judge
- Regression testing
- CI
New to this? Do the toolkit setup first, then add these one at a time.
- 1
Config-driven LLM tests with assertions and side-by-side prompt comparison.
$ npx promptfoo@latest init
$ npx promptfoo@latest eval
- 2
- 3
How to show it to an employer
1Screenshot of the red check on a bad change
2README with the rubric, thresholds and why
3A before-and-after table for one real prompt improvement
Stretch goals
Track cost and latency per case
Check how often the LLM judge agrees with your own labels
Store results over time and chart the trend
Questions this prepares you for
How do you evaluate an LLM feature without a single right answer?
When do you trust an LLM judge?
What goes in a golden set?
Practise them in a mock interview Next projectTool-calling agent with guardrails→