AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel Husain

This post is auto-generated from a public YouTube video for personal study notes. It is not an endorsement. The transcript is machine-derived and may contain errors; refer to the original video for accuracy.

About this video

​ ​ Today, I want to share a new episode with Hamel Husain.

Hamel has trained 2,000+ PMs and engineers from companies like OpenAI, Anthropic, and Google on how to run AI evals.

In my new episode, he shares a free master class on how to build evals for a real AI agent in just 50 minutes using a simple spreadsheet. I learned a lot from Hamel and I think you will too.

Hamel and I talked about: (00:00) What the most valuable part of evals is (01:25) Live walkthrough: Analyzing 100 real production traces (09:50) Creating the eval criteria using a simple spreadsheet (24:44) Why binary pass/fail ratings beat 1-5 scores every time (28:52) The agreement metric trap that fools most PMs (30:08) True positive and negative rates explained (36:00) How to set up continuous evals in production

Where to find Hamel: