Why framing comes before models
In Level 1 you saw that LLM features are probabilistic: the same input can produce different outputs, and "correct" is often a judgement call. That makes framing more important, not less. If you do not decide what success looks like before anyone writes a prompt, the team will end up arguing about whether a demo "feels good", and that argument has no end.
This lesson follows our running feature, an AI email triage assistant that sorts incoming mail for busy professionals. By the end you will have produced a one-page Problem and Success Brief with three parts:
- A problem statement
- Three to five success criteria with target thresholds
- A decision on the right tool: rules, a simple heuristic, or an LLM
Task 1: Write the problem statement
A good problem statement names the user, the job they are trying to get done, and the current pain. The "job" idea comes from Clayton Christensen's Jobs to Be Done theory: customers "hire" a product to help them make progress toward a goal. Your problem statement should describe that progress. It should not describe your solution.
Do this:
- Name a specific user. "Everyone with an inbox" is too broad. Try: "Account managers at B2B software companies who handle 150+ inbound emails a day."
- State the job. What progress do they want? "Spot the emails that need my action today, so that no client request slips."
- Describe the current pain, with evidence. Use interviews, support tickets or usage data. Industry research points the same way: a 2012 McKinsey study estimated that interaction workers spend about 28% of the workweek reading and answering email. Treat figures like this as background. Your own users' data is what counts.
- Check for solution words. If the statement contains "AI", "LLM" or "model", rewrite it. The problem has to stand on its own.
Example:
Account managers handling 150+ emails a day need to find the messages that require action today. They currently scan the whole inbox two or three times a day, which takes about 45 minutes, and they still miss roughly one urgent client request a week.
The numbers in this example are illustrative. Replace them with your own baseline measurements.
Task 2: Define three to five success criteria
A success criterion is a measurable outcome with a target threshold. "Users find it helpful" is not a criterion. "At least 90% of emails labelled Urgent are truly urgent" is one.
Do this:
- Derive each criterion from the problem statement. Every pain you named should map to a metric.
- Set a baseline first. Measure the current state, for example 45 minutes of scanning a day, before you set a target.
- Write a threshold for each criterion. Use a number, a direction and a timeframe.
- Add at least one "do no harm" criterion. It catches the failures that would destroy trust, even when the headline metric looks good.
- Stop at five. Any more and the team will not know which ones matter.
Example criteria for the triage assistant:
- Priority label accuracy: At least 90% of emails get the label (Urgent / Today / Later / FYI) that the user would have chosen.
- Missed-urgent rate (do no harm): No more than 1 in 100 truly urgent emails is labelled Later or FYI.
- Time saved: Daily inbox scanning time falls from about 45 to 20 minutes or less.
- User agreement: Users change no more than 15% of labels.
The missed-urgent rate matters because the two error types are not equal. Labelling a newsletter Urgent is a small annoyance. Burying a client's escalation in FYI can lose a deal. Decide which errors are costly now. Later levels will turn these criteria into a full spec and an eval set.
Task 3: Decide whether an LLM is the right tool
Google's Rules of Machine Learning opens with Rule #1: "Don't be afraid to launch a product without machine learning." A simple heuristic often gets you most of the value and gives you a baseline to measure any model against.
Do this. Go through the parts of the job one at a time:
- List the decisions the feature must make. Examples: "Is this from a key client?" "Is it a newsletter?" "Does it ask me to do something by a deadline?"
- For each decision, ask whether a rule could make it reliably.
- Known sender lists, domains, mailing-list headers → rules
- Patterns in the user's own behaviour (who they reply to quickly) → simple heuristic or classic ML. Gmail's Priority Inbox, launched in 2010, used signals like this, such as who you email most often.
- Free-text meaning ("Can you send the revised quote before Friday's board meeting?") → an LLM is a good fit
- Weigh the trade-offs. Compared with rules, LLMs add cost per call, latency, variable output and privacy questions about sending email content to a model.
- Pick the simplest mix that meets your thresholds. A common answer is a hybrid: rules handle the clear-cut cases, and the LLM handles only the ambiguous remainder.
- Record the decision and the reason in your brief.
Red flags that you are reaching for an LLM too early:
- The decision can be written as a short if/then list
- You need 100% predictable output, for example for compliance routing
- You have no criteria yet to tell you whether the LLM is better than a rule
Your artefact: the Problem and Success Brief
Put together a single page containing:
- Problem statement (user, job, pain, with evidence)
- Baselines for each pain you named
- 3–5 success criteria with thresholds, including one do-no-harm criterion
- Tool decision (rules / heuristic / LLM / hybrid) with your reasoning
Share it with your engineering and data science partners before anyone starts on prompts.
Key takeaways
- Frame the problem before you pick a model: name the user, the job to be done and the current pain, without using solution words.
- Success criteria need numbers: a baseline, a target threshold and a direction.
- Include at least one do-no-harm criterion, because the different kinds of error do not cost the same.
- Keep to three to five criteria so the team knows what matters.
- Start with the simplest tool that works. Rules and heuristics give you a baseline, and a hybrid often beats an all-LLM design.