Why a profile comes before a product
Every AI request lands on your desk in vague language: "We want a fast, cheap assistant that summarises claim emails." If you go straight to a service, you end up choosing by habit, or by whichever demo you saw last. A workload profile is a one-page record of what the job really needs, written before anyone names a product. Think of it as the non-functional requirements sheet you would write before choosing a database, extended with the questions AI raises: which model, what it costs per token, and how you prove it is safe.
The rule that matters most is that the profile is service-neutral. It never says "use Foundry" or "use Azure ML". Later lessons score each service against it, and that scoring is only fair if the profile doesn't already lean towards an answer.
The twelve fields
# workload-profile.yaml — one per workload, kept in the repo next to the design
name: claims-email-summariser
task_type: generative # generative | predictive | perception
build_path: buy-api # buy-api | adapt-model | train-from-scratch
data:
sensitivity: confidential # public | internal | confidential | regulated
residency: EU # where prompts may be PROCESSED, not just stored
latency:
p95_seconds: 8 # 95% of requests finish within 8 s
interactive: false # does a human wait on the screen?
throughput:
peak_requests_per_min: 50
peak_tokens_per_min: 90000
volume_and_cost:
monthly_requests: 440000
monthly_budget_usd: 3500
model_choice:
lock_in_tolerance: medium # could we swap model family in a sprint?
evaluation_and_safety:
quality_metric: "adjuster rates summary usable >= 90%"
safety: [content filtering, PII handling, human review of payouts]
mlops_maturity: low # none | low | medium | high
accelerators: none # none | inference GPU | training GPU cluster
team_skills: [python, REST, no data scientists]
cost_model: pay-per-token # pay-per-token | reserved capacity | per-request | compute-hours
A few fields need explaining.
Task type. Generative means producing text, code or images (an LLM). Predictive means scoring or forecasting from tabular data: churn, fraud, demand. Perception means reading the world: speech to text, OCR, image tagging. Choosing the wrong one here is the costliest mistake in the profile. Plenty of "we need GenAI" requests turn out to be a predictive model on a spreadsheet.
Build path: buy, adapt or train. Go up this ladder only when you have evidence that you need to:
| Path | Pick it when the profile shows… | Typical evidence |
|---|---|---|
| Buy an API | The task is common and quality is acceptable straight away | A 50-sample pilot scores above the quality bar |
| Adapt a model | General models miss your style, labels or domain terms | The pilot fails on jargon; you have hundreds or thousands of labelled examples |
| Train from scratch | Your data is unique, tabular or proprietary, and no pretrained model applies | Predictive task on your own history, e.g. 5 years of claims outcomes |
Each step up adds data work, compute and MLOps load. A team with mlops_maturity: low and no data scientists should take a lot of convincing before it leaves "buy".
Residency. Record where data may be processed, not only where it is stored. Azure OpenAI deployment types show why this matters: Global types can process prompts in any geography where the model is deployed, Data Zone types stay inside a Microsoft-defined zone (US or EU), and regional types process in the deployment's region. Data at rest stays in your chosen geography in every case. If you write "EU" in the profile, you have already ruled some of these options out.
Turning "fast" and "cheap" into numbers
Adjectives can't be tested. Ask three questions: how many, how quickly, and how much per month?
# sizing.py: turn interview answers into profile numbers
emails_per_day = 20_000 # from the claims team's mailbox stats
working_days = 22
peak_hour_share = 0.15 # 15% of a day's mail arrives in the busiest hour
in_tokens = 1_500 # avg email + instructions (measure a sample!)
out_tokens = 300 # target summary length
peak_rpm = emails_per_day * peak_hour_share / 60 # 50 requests/min
peak_tpm = peak_rpm * (in_tokens + out_tokens) # 90,000 tokens/min
monthly = emails_per_day * working_days # 440,000 requests
in_m = monthly * in_tokens / 1e6 # 660 M input tokens
out_m = monthly * out_tokens / 1e6 # 132 M output tokens
# ILLUSTRATIVE rates only: look up today's price list before deciding
price_in, price_out = 2.50, 10.00 # USD per 1M tokens
cost = in_m * price_in + out_m * price_out
print(peak_rpm, peak_tpm, round(cost)) # 50.0 90000.0 2970
Now "cheap" means under $3,500 a month, and "fast" means p95 under 8 seconds.
Use percentiles, not averages. An average of 2 s can hide a p99 of 30 s, and the slowest requests are the ones users complain about. For chat, also record time to first token, because users read a streamed answer as it arrives.
It's worth asking whether anyone actually waits for the answer. In this case adjusters open a claim hours after the email arrives, so interactive: false. That one answer opens up a bigger cost lever. Azure OpenAI's Global Batch option runs asynchronous jobs with a 24-hour target turnaround at 50% less than Global Standard, which would bring the example to about $1,485. The profile captured the fact that unlocks the saving. Choosing the actual mechanism comes later.
Gotcha: output tokens usually cost several times more than input tokens. Capping the summary length is often the cheapest optimisation you'll find.
Cost model and lock-in
Write down how the money is spent, as well as how much. Common shapes are pay per token, reserved capacity (for example Azure's provisioned throughput, billed hourly per unit deployed whether you use it or not, with monthly or yearly reservations for discounts), per-request or per-page pricing for ready-made APIs, and compute-hours for training and hosting your own models. Spiky, uncertain volume suits pay-per-use. Steady, high volume can justify reserved capacity.
For lock-in, record how painful a model swap would be. If prompts, evaluation sets and safety tests are portable, tolerance can be high. If you fine-tune a vendor's model, you are tied to that model.
A worked mini-profile: predictive
Request: "Flag claims likely to be fraudulent." The task type is predictive, the build path is train (on five years of labelled outcomes), sensitivity is regulated, latency is p95 under 300 ms at 20 requests a minute, and the cost model is compute-hours for training plus an always-on endpoint. Evaluation needs precision and recall plus a fairness check across customer groups. With MLOps maturity low, the profile has already shown the main risk: retraining and drift monitoring, not the model itself.
Key takeaways
- The profile has twelve fields and names no service. Scoring services against it comes later.
- Classify the task type first: generative, predictive or perception.
- Move from buy to adapt to train only when pilot results and labelled data justify it.
- Record where data may be processed, not just stored.
- Replace adjectives with numbers: p95 latency, peak requests and tokens per minute, and a monthly budget.
- Ask whether a human waits for the answer. A "no" opens up batch-style savings.
- Record the cost model (per token, reserved, per request, compute-hours) as well as the total.