AI ENGINEERING
Week 2 · Foundations

Evals & Engineering Discipline

Make it good, prove it

⏱ ~10 hours, hands-on🐍 Python / 🔷 TypeScript🛠 You ship: An agent you can measure and improve
What this week is

Last week you built an agent from scratch and you know exactly how it works. This week you learn the part of the job that separates a demo from a product: the discipline.

We build a new agent — a diagram design assistant that draws flowcharts, org charts and architecture diagrams onto a canvas you can watch in your browser — and then we stop adding features and start measuring. Golden datasets. Automated scorers. A baseline.

Then, for the entire second half, we improve the agent one technique at a time — and after every single change we re-run the evals and look at the numbers. Some scores will jump 2×. Some won't move, and that's a lesson too.

You never ship an improvement you haven't measured. That single habit is the profession.

The walkthrough

Each part adds one capability to the same growing codebase, and each is a runnable checkpoint.

  1. 01Build the diagram agentA domain agent with a visible canvas, so every improvement is something you can see as well as score.
  2. 02Golden datasetsCurate the cases that matter, including the ones your agent currently fails.
  3. 03Automated scorersDeterministic checks where possible, LLM-as-judge where not — and knowing which is which.
  4. 04Establish a baselineThe number you have to beat. Without it, every later change is an opinion.
  5. 05Context engineeringChange what the model sees, not which model it is, and re-measure.
  6. 06Sharper tool designRewrite descriptions and schemas; watch the score move without touching the loop.
  7. 07RAG from scratchGround the agent in real reference material and measure whether it actually helped.
  8. 08Generative UIRicher output than text — and the evals that keep it honest.
  9. 09Human-in-the-loopApproval as a product feature, not just a safety valve.
  10. 10Smarter architecturesRestructure the agent and let the numbers arbitrate.
  11. 11The data flywheelEvery user correction becomes a new test case, so the suite grows with usage.
  12. 12Ship itA final measured comparison against the Part 4 baseline.

After this week you can

  • Build golden datasets and automated scorers for a real agent
  • Improve an agent with evidence instead of intuition
  • Know when RAG helps and when it is overhead
  • Turn user corrections into a growing regression suite