Week 6 · Tue, Oct 20, 2026 · Evals, guardrails, and hackathon prep · Taught by Fard Johnmar
Test your AI before you trust it
We ran the same 20 civic prompts through three models, three times each, to see which differences were real and which were noise. Then we added guardrails and set what "done" means for Demo Day.
What we covered
Right, Safe, Steady
Right = is the answer correct? Safe = did it stay inside the guardrails? Steady = is it just as good on runs 2, 3, and 20?
One run isn't proof
Agents give different outputs for the same input. Run it several times before you believe a score.
Most "wins" are noise
A small gap between two models is often just jiggle. Compare them properly before you switch.
Clean going in, checked coming out
Strip personal info before the model, label retrieved text as untrusted, and validate output before anything runs.
Keep the architecture simple
Start with one agent. Add a checker before you add a second worker.
Add one eval to your hackathon idea
About 60 minutes. A browser. The lab runs in Colab with no install, no API key, and no cost.
- Pick the scariest failure: what would embarrass you on stage? Is it a Right, Safe, or Steady problem?
- Write 5 to 10 test cases: one easy, one hard, one that tries to break it. Synthetic data only.
- Pick the cheapest scorer that works: contains or exact match, then schema check, then rubric, then LLM judge.
- Set your threshold before you run it, for example 9 of 10 Right and 10 of 10 Safe.
- Run it 3 times and write down all three scores.
- Fill in homework/eval-template.md, then submit it in Google Classroom with your eval and a screenshot of your 3 runs.
- Stuck? Open the repo with your coding agent and say "Help me with the Week 6 homework."
Fellows turn in homework through Google Classroom. The Classroom link is on the last slide.
Want the full fellowship?
Classes are open to everyone. Fellows also get homework feedback in Google Classroom, a cohort Discord role, office hours, and priority seats at the hackathon.