AI Product Management: Specs, Evals and Launches

What's inside

1. Level 1: How AI Products Differ from Deterministic SoftwareFree
   Learners see why LLM features give probabilistic, variable outputs, and how that changes the way a PM defines, tests and ships the AI email triage assistant.
2. Level 2: Framing the Problem and Defining Success CriteriaFree
   Learners define the user problem behind email triage and turn it into measurable success criteria before any model work starts.
3. Level 3: Writing an AI Spec with Examples and Failure Modes
   Learners write an AI product spec that defines expected behaviour through worked input-output examples, a list of failure modes and how each failure should be handled.
4. Level 4: Building Eval Sets: Golden, Edge and Adversarial
   Learners build a structured eval set that combines representative golden examples, edge cases and adversarial inputs drawn from the spec.
5. Level 5: Scoring Methods: Human Ratings, Rubrics, LLM-as-Judge
   Learners compare human ratings, rubric-based scoring and LLM-as-judge for scoring eval outputs, including the biases and limits of each.
6. Level 6: Setting Quality Bars and Making Go/No-Go Calls
   Learners turn eval scores into explicit quality bars, including hard blockers for severe failures, and use them to make go/no-go decisions.
7. Level 7: Offline Evals versus Online Metrics
   Learners link offline eval results to production metrics such as task success, acceptance rate, edits, escalations and retention.
8. Level 8: Human-in-the-Loop Review and Sampling
   Learners design an ongoing human review process that samples production outputs, labels them and feeds the findings back into evals and the spec.
9. Level 9: Guardrails, Safety and Red-Teaming in Outline
   Learners outline the input and output guardrails and the red-teaming exercises needed to launch the triage assistant safely.
10. Level 10: Versioning Prompts and Models
    Learners set up versioning for prompts, model identifiers and configuration so that every output can be traced and every change can be rolled back.
11. Level 11: A/B Testing LLM Features
    Learners design A/B tests for LLM features that account for variable outputs, novelty effects and the need for guardrail metrics.
12. Level 12: Cost and Latency Budgets and Model Choice
    Learners set cost and latency budgets for each request and compare models and architectures against quality, cost and speed.
13. Level 13: Handling Model Upgrades and Regressions
    Learners run a controlled process for evaluating, rolling out and if needed reverting model upgrades, whether chosen by the team or forced by a vendor deprecation.
14. Level 14: Incidents and Protecting User Trust
    Learners handle AI-specific incidents, from detection and triage through fix, user communication and post-incident review.
15. Level 15: Privacy and Data Use Rules
    Learners set rules for how email content is processed, stored, retained, reused for evals or training, and shared with model vendors.
16. Level 16: Working with Engineering, DS, Design, Legal, Support
    Learners set clear roles, rituals and handoffs with each partner function across the lifecycle of the AI feature.
17. Level 17: Roadmapping When Capabilities Change Monthly
    Learners build a flexible roadmap that plans around fast-changing model capabilities, costs and vendor offerings without losing strategic focus.
18. Level 18: Final Challenge: Spec to Post-Launch Review Exam
    Learners apply the whole course by taking the triage assistant from problem framing and spec through evals, launch, monitoring and a post-launch review.

Access

The first 2 levels are free with a free account. Every level, the podcast edition and the AI tutor come with All Access at £4.99/month or any Creator plan — see pricing.

Start free →