Lesson 2: Choosing a Model: Quality, Speed and Cost

Why model choice is an engineering decision

Picking a model is like picking an instance size in the cloud. The biggest one always works, but you pay for it on every request and your users wait for it. The smallest one is cheap and quick, but it may get your task wrong. The job is to find the smallest model that reliably does your task. You find it by measuring on your own prompts, not by reading leaderboards.

You weigh four things:

• Quality: does it get your task right, every time?
• Latency: how long the user waits for the answer.
• Context window: how much text fits into one request.
• Price: dollars per million tokens, charged separately for input and output.

Step 1: See what your project can actually call

The public model list and your project's access are not always the same thing. Ask the API directly, using the key setup from Level 1 (OPENAI_API_KEY in your environment, never in code):

# list_models.py: pip install openai
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

for m in client.models.list():          # GET /v1/models
    if m.id.startswith("gpt-"):         # skip embeddings, audio, etc.
        print(m.id, m.owned_by, getattr(m, "shutdown_date", None))

// list_models.mjs: npm install openai
import OpenAI from "openai";
const client = new OpenAI();
for await (const m of client.models.list()) {
  if (m.id.startsWith("gpt-")) console.log(m.id, m.owned_by);
}

Gotcha: each model object gives you only an id, a created timestamp, owned_by, and a shutdown_date (null if no shutdown has been announced). It does not give you the context window, modalities or price. You get those from each model's page and from the pricing page in OpenAI's developer docs.

Step 2: Read the spec sheet

Here are three current flagship models as listed in October 2026, plus one older model that has a dated snapshot. Standard tier, US dollars per 1M tokens:

All four take text and image input and return text. Astra costs 100× as much as Luna per token. That gap is the reason to measure before you choose.

Pitfalls on the pricing page: Output tokens cost about 5× as much as input tokens, so a model that writes long answers costs more than its input price suggests. Requests over 272K input tokens are billed at higher long-context rates. The Batch API costs 50% less (covered in a later level). These numbers will change. Treat this table as a dated snapshot and check the live page before you budget.

Step 3: Benchmark on your own prompt

Leaderboards measure someone else's task. Take a real prompt from your app, run it against two or three models, and record what happens. We call client.responses.create here and leave its details to Level 3.

# compare.py
import time
from openai import OpenAI

client = OpenAI()

# Prices per 1M tokens, copied from the pricing page. Write the date down!
PRICES = {  # model: (input, output)
    "gpt-6-luna": (0.10, 0.50),
    "gpt-6.1-sol": (2.00, 10.00),
    "gpt-5.4-2026-03-05": (2.50, 15.00),
}

PROMPT = ("A customer writes: 'I was charged twice for order 4471 and "
          "your bot keeps closing my ticket.' Classify urgency "
          "(low/medium/high) and draft a two-sentence reply.")

print(f"{'model':<22}{'secs':>6}{'in':>7}{'out':>7}{'reason':>8}{'cost 



:>11}")
for model, (p_in, p_out) in PRICES.items():
    t0 = time.perf_counter()
    r = client.responses.create(model=model, input=PROMPT)
    secs = time.perf_counter() - t0          # wall-clock latency

    u = r.usage                               # token counts come back with every response
    reasoning = u.output_tokens_details.reasoning_tokens
    cost = u.input_tokens * p_in / 1e6 + u.output_tokens * p_out / 1e6
    print(f"{r.model:<22}{secs:>6.1f}{u.input_tokens:>7}"
          f"{u.output_tokens:>7}{reasoning:>8}{cost:>11.6f}")

A run might print something like this (illustrative; your numbers will differ):

model                   secs     in    out  reason     cost $
gpt-6-luna               1.2     62    180      40   0.000096
gpt-6.1-sol              6.8     62    650     470   0.006624
gpt-5.4-2026-03-05       2.1     62    210       0   0.003305

How to check your result: do the arithmetic for one row by hand (62 × $0.10/1M + 180 × $0.50/1M ≈ $0.000096), then read the answers. If Luna classifies the ticket correctly and writes a decent reply, it wins by a factor of about 70 on cost. Run each prompt 5–10 times and on several different prompts. A single run tells you little.

Step 4: When is a reasoning model worth it?

Reasoning models think before they answer. That thinking is billed as hidden reasoning tokens, which count as output tokens and use up context window. You can see them in usage.output_tokens_details.reasoning_tokens. You control how much thinking happens with reasoning={"effort": "low"}. Supported levels vary by model, from none up to max.

• Worth it: multi-step logic, tricky debugging, planning, maths, and anything where one wrong answer costs more than the wait.
• Not worth it: classification, extraction, rewriting, chat replies, and high-volume pipelines where latency matters.

Rule of thumb: start with the cheapest model at low effort. Move up only when your test prompts show it failing.

Step 5: Pin the model you tested

An alias such as gpt-5.4 can be moved to a newer version without notice. A dated snapshot such as gpt-5.4-2026-03-05 stays fixed. In production, use the exact model you benchmarked, so a silent upgrade can't change your outputs.

MODEL = "gpt-5.4-2026-03-05"   # pinned: same behaviour until you choose to change it

The newest models, such as the GPT-6 family, currently list only a single ID with no dated snapshot. For those, log response.model on every call and re-run your comparison script whenever OpenAI announces an update. Also watch shutdown_date: a pinned snapshot is retired eventually, and the date tells you when you have to move.

Key takeaways

• GET /v1/models shows what your project can call. Context window, modalities and price come from the docs.
• You pay separately for input and output tokens. Output costs several times more, and reasoning tokens are billed as output.
• Measure latency, tokens and cost on your prompts, several runs each. Leaderboards don't measure your task.
• Use a reasoning model when correctness on hard, multi-step problems is worth the extra time and money.
• Pin a dated snapshot where one exists, log response.model, and keep an eye on shutdown dates.