Lesson 2: The Standard AI Workload Profile

Why write a profile before picking a service

Level 1 gave you the map: the AWS AI and ML services, grouped. A map does not tell you which road to take. Before you choose an EC2 instance family you write down CPU, memory, network and budget. An AI workload needs the same kind of requirements sheet. We'll call it the workload profile.

The profile is written once and reused. Every service lesson from here on (Bedrock, SageMaker AI, Textract and the rest) is scored against it, and so is the final cross-cloud scorecard. A fuzzy profile leads to a fuzzy decision. If the profile is full of numbers, the choice almost makes itself.

The fields

Step 1: Classify the task

• Generative: produces new content (text, code, images, summaries). Example: drafting email replies.
• Predictive: learns a number or label from your historical, usually tabular, data. Examples: churn risk, fraud score, demand forecast.
• Perception: turns the messy world into structured data. Examples: reading invoice fields, transcribing calls, spotting objects in photos.

Then choose how much model work you take on:

• Buy: call a ready-made API and do no model work. Invoice fields go to a document API and call audio goes to speech-to-text.
• Adapt: take an existing foundation or pre-trained model and steer it with prompts, retrieval of your documents, or fine-tuning.
• Train: build your own model on your data. You do this when the signal exists only in your data, as with churn from your own billing history.

Best practice: start at buy and move right only when you can write down why. Every step right adds data work, evaluation work and MLOps you must staff.

Step 2: Turn vague requirements into numbers

The brief you'll actually get sounds like this: Draft replies to customer emails. Make it fast and cheap, and keep the data in the UK. None of that can be scored yet. Ask for volumes and do the arithmetic in code:

# Inputs gathered from the support team's ticket system
tickets_per_day   = 20_000
input_tokens      = 1_500   # email thread + instructions, measured on 50 samples
output_tokens     = 300     # typical drafted reply
peak_hour_share   = 0.20    # 20% of daily tickets land in the busiest hour

# Volume
daily_in  = tickets_per_day * input_tokens    # 30,000,000
daily_out = tickets_per_day * output_tokens   #  6,000,000
peak_rpm  = tickets_per_day * peak_hour_share / 60   # ~67 requests/min

# Cost: ILLUSTRATIVE prices only, replace with today's price list
price_in_per_m, price_out_per_m = 3.00, 15.00   # USD per 1M tokens
daily_cost   = daily_in/1e6*price_in_per_m + daily_out/1e6*price_out_per_m
monthly_cost = daily_cost * 30
print(round(peak_rpm), daily_cost, monthly_cost)
# -> 67  180.0  5400.0   (inside a $6,000 ceiling, but only 10% headroom)

There is a second number to check: quota. On Bedrock, on-demand throughput is limited mainly by a per-model tokens per minute quota, which counts input and output together. Some models count each output token several times against that quota; current Anthropic Claude models use 5x, 10x or 15x depending on version. The quota is also reserved up front as input tokens plus max_tokens, and the unused part is returned when the request finishes. You are billed only for the tokens actually used.

burndown   = 5          # check the rate for YOUR model; the default is 1:1
max_tokens = 4_096      # a lazy default
reserved_tpm = peak_rpm * (input_tokens + max_tokens)        # ~373,000 reserved at the start
settled_tpm  = peak_rpm * (input_tokens + output_tokens*burndown)  # ~200,000 after settling
# Setting max_tokens near 400 cuts the up-front reservation to ~127,000 TPM

Gotcha: teams hit throttling at a fraction of their expected volume because max_tokens was left at a large default. Size it to the outputs you actually get.

Step 3: Pin down latency and residency precisely

Latency. Write targets as p95, never as an average. The slowest 5% are the users who complain. Also say which latency you mean. Bedrock's CloudWatch metric InvocationLatency runs from the request to the last token. TimeToFirstToken runs to the first token on streaming calls. For a chat UI the first one matters most, and for a batch job only the total does. Once a pilot runs, check the target against real data:

# p95 end-to-end latency per hour for one model during the pilot
aws cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock --metric-name InvocationLatency \
  --dimensions Name=ModelId,Value=<your-model-id> \
  --extended-statistics p95 --period 3600 \
  --start-time 2026-10-28T00:00:00Z --end-time 2026-10-29T00:00:00Z

Residency. You need to know where data is processed, not only where it is stored. Bedrock's geographic cross-Region inference (for example an eu profile) can send your prompts and outputs to other Regions inside that geography over AWS's encrypted network, while stored data stays in the source Region. Global cross-Region inference can route to any supported commercial Region. If your rule is "UK only" rather than "EU is fine", record that exactly, because it removes options later.

Step 4: Write the profile down

workload: support-reply-drafter
task_type: generative
approach: adapt            # prompt + company knowledge; no training data exists
data: {sensitivity: customer PII, residency: EU processing acceptable, home_region: eu-west-2}
latency: {p95_time_to_first_token_ms: 1500, p95_total_ms: 8000}
throughput: {peak_requests_per_min: 70, peak_tokens_per_min_reserved: 130000}
volume: {input_tokens_per_day: 30000000, output_tokens_per_day: 6000000}
lock_in: must swap model within one sprint
evaluation: 200 graded past tickets; PII filter on output
mlops_maturity: none
accelerators: none (managed API)
team: 3 backend developers, no data scientist
cost: {ceiling_usd_month: 6000, model: pay-per-use, prices_dated: 2026-10}

Keep it in the repo next to the code. Later lessons score each service against these exact lines.

Key takeaways

• A workload profile is a reusable requirements sheet. Every later service choice is scored against it.
• Classify the task (generative, predictive or perception) and the approach (buy, adapt or train). Start at buy and give a reason for each step towards train.
• Convert words into numbers: tokens or requests per day, peak per minute, a p95 latency target, a residency rule and a monthly budget ceiling.
• Bedrock quota is reserved as input plus max_tokens, and some models count each output token several times. Plan throughput with these numbers, not with your bill.
• Name the latency you mean (first token or last token), and state residency as where data is processed.
• Prices are dated examples. Record the date next to each one.