AI Evals for Practitioners: Proving Your AI Actually Works

Topics covered

• •Explain why informal "vibe checks" stop working once an LLM feature has real users and frequent changes
• •Define an eval precisely (input, system under test, grader, verdict) and tell product evals apart from model benchmarks
• •Describe the three levels of evaluation (code assertions, human and model review, A/B tests) and how often each runs

Topics covered

• •Collect and sample traces so that a review covers real variety rather than happy paths
• •Open-code traces by writing short, factual notes about the first failure in each one
• •Turn open codes into a failure taxonomy through axial coding, using an LLM only as a reviewed assistant

Topics covered

• •Design a golden dataset whose cases carry inputs, fixtures, expected behaviour and tags
• •Generate realistic synthetic inputs with dimensions and tuples rather than one-shot "give me test questions" prompts
• •Cover edge cases, negative cases and unanswerable questions so that the set tests both what the system should and should not do

Topics covered

• •Choose the cheapest reliable grader for each failure mode: a code check, an LLM judge or a human
• •Write a binary judge prompt for one failure mode, with definitions, few-shot examples and a critique before the verdict
• •Validate a judge against human labels using train, dev and test splits and the true positive and true negative rates

Topics covered

• •Separate retrieval failures from generation failures before choosing any metric
• •Measure retrieval with Recall@k, Precision@k and MRR on a question-to-chunk dataset
• •Evaluate generation for faithfulness and answer relevance, and know what RAGAS-style metrics do and do not measure

Topics covered

• •Build a regression suite that runs on every prompt, model, retrieval or tool change and blocks known regressions
• •Set up online monitoring that samples live traces, applies validated evaluators and feeds new failures back into the suite
• •Distinguish guardrails from evaluators and decide which checks may run inline