Notes from Hamel Husain's experience designing eval interviews

A guardrail can pass every example in a dataset and still fail at its job. If every input is allowed, a system that accepts everything gets a perfect score. The test never asked it to reject anything.

That was the example that stayed with me from Hamel Husain's Cracking the AI Evals Interview. He has helped design these interviews, and his advice puts the data before the score. Before improving a system, establish whether the evaluation can tell you anything useful about it.

An eval is a test of whether an AI system does the job you need. That includes choosing examples, deciding what counts as success, and checking the results. In an interview, each of those choices gives the interviewer evidence of how you think.

TLDR

1. Inspect the dataset before trusting its scores. Missing labels and missing failure cases can invalidate the evaluation.
2. Start experiments with a simple baseline, then explain what each change tests.
3. Connect the results to a product decision, including cost, latency, and remaining failures.
4. State the limits of your evidence and propose the next useful experiment or product change.
5. Make your use of AI tools clear. You still own the reasoning and the recommendation.

Engineering and data science roles are colliding

During the Q&A, someone asked whether the approach was for data scientists and ML engineers or relevant to other roles. Hamel described evals as a discipline with a strong data science foundation, then said that "roles are collapsing." He also made the scope clear: you don't need to learn all of data science before you can learn evals. That exchange starts around 33:32 in the recording.

I think this is the most useful career observation in the session. Software engineers building AI applications are taking on work that overlaps with data science. We need to inspect labels, design comparisons, and ask whether a result holds on examples we haven't used for tuning. An endpoint returning successfully tells us little about whether its advice was sound. Owning the feature increasingly means owning the evidence that it works.

Hamel's example is TreeBuddy, a hypothetical application that takes a photo of a tree, identifies it, and offers health and care recommendations. It makes the overlap concrete. You can implement the application without being a tree expert, but you can't assume you're qualified to invent missing health labels. Engineering skill helps you build the test; domain knowledge helps establish what the right answer is.

Seven sections that make the reasoning visible

The outline below follows the seven sections on Hamel's slide. Each section answers a different question for the person reviewing your work.

1. Executive summary

Give the reviewer the decision in one paragraph. State what you recommend, the evidence that supports it, and the limitation that could change the decision. For TreeBuddy, that could mean recommending a configuration for species identification while explaining that the available labels don't support a conclusion about health advice. Write this after the experiments, even though it appears first. A reviewer should be able to understand your recommendation without reconstructing it from a notebook. The summary is also a useful check on your own work: if you can't explain what the results justify, the experiment hasn't yet produced a clear decision.

2. Data

Open the examples and inspect them yourself. Hamel's TreeBuddy exercise includes uneven label coverage, heights with unclear units, and a guardrail dataset with no disallowed inputs. Those problems change what a score means before you touch the prompt. Summarize what you found and how you handled it. Synthetic examples can help test obvious input violations, while external labeled data may help with missing coverage if it matches the task. Neither gives you permission to guess specialist labels. Hamel recommends spending roughly 50% to 70% of the assignment on data inspection, a guideline for this kind of interview rather than a universal development budget.

3. Experiments

For each experiment, state the goal, what you tried, and the methods and findings. Start with the simplest reasonable baseline, such as one model with a prompt. Then test a change for a reason: does retrieving relevant tree information improve identification, or does a different model help on the same examples? Retrieval means supplying the model with information found in an external source. Keep examples used for tuning separate from those reserved for final evaluation. Explain failures too. An improvement on familiar examples that disappears on held-out examples should make you question how well the approach generalizes. A more elaborate setup needs evidence that its added complexity is useful.

4. Results

Bring the experiments together into a recommendation. Show the baseline and the final scores for the behaviors you measured, alongside cost and response time. Explain which failures remain. Hamel emphasizes that you can demonstrate good judgment by rejecting a complicated approach even when it scores higher. The extra quality may not justify its cost or delay for the product you're building. A score table becomes useful when the reader can connect it to that choice. Also keep the scope honest: a species identification result doesn't establish that the same system gives reliable care recommendations. Those are different behaviors and need their own evidence.

5. What I learned

Explain what the experiments changed in your understanding. Perhaps retrieval helped, prompt tuning didn't, or the evaluation data was too sparse to support a conclusion. This is also where you can explain a weak requested metric diplomatically. Hamel gives the example of grading helpfulness and tone from one to five, then combining them into a score out of 100 without clear criteria. Compute what the assignment requires, but explain what you would measure differently and why. Specific checks, such as whether an answer blames the user, make a failure easier to act on. A pass/fail label still needs a clear definition and a trustworthy evaluator, as his evals guide explains.

6. Next steps

Choose the next action based on the uncertainty or failure you found. If health labels are missing, better expert-reviewed data may be more useful than another model comparison. If the photo doesn't show enough of the tree, the application could guide the user to take a better picture before offering advice. Hamel uses that example to show how evals can inform product design. The fix may belong in the input flow. Time-box the assignment, then explain what you'd investigate next and what result would change your recommendation. A next-steps section should give the team a useful decision to make after reading your report.

7. AI tools used

The final item on Hamel's outline is AI tools used. My practical interpretation is to make the division of work explicit. Describe where a coding assistant helped, whether you generated synthetic examples, and how you checked the outputs you relied on. Distinguish a model helping write evaluation code from a model grading the application's answers. The second use makes that model part of the measurement system and needs validation against human judgment where judgment is required. An assistant can organize results and help implement an experiment. You still need to explain why the dataset fits the task and why the evidence supports your recommendation.

Own the decision the eval supports

My background in test automation makes the missing guardrail cases feel familiar. A passing test has always depended on what it actually checks. With AI applications, we also have to decide whether the examples and grading criteria represent the behavior users need.

That responsibility is where engineering and data science meet. If I own an AI feature, I need to be able to explain what the evaluation establishes, what it leaves uncertain, and what I would do with that uncertainty. The useful interview deliverable is a recommendation another engineer can inspect and act on.

References and further reading

Hamel Husain: Cracking the AI Evals Interview. The TreeBuddy exercise, seven-section report outline, and Q&A about overlapping roles. The outline appears around 05:57; the roles discussion starts around 33:32.

Hamel Husain and Shreya Shankar: AI Evals: Everything You Need to Know. Further guidance on data review, failure-specific checks, and validating evaluators.