Lesson 1: The Workload Decides, Not the Chip

The accelerator myth

You have a slow Python job, and someone says "put it on a GPU". Sometimes that gives a 50x speed-up. Often you get 1.05x and a bigger cloud bill. The chip isn't the reason. What decides it is where the job spends its time, and you only find that out by measuring.

Think of a kitchen. If dinner takes an hour and 40 minutes of that is waiting for deliveries, buying a faster oven won't get you much. This level teaches you to time the deliveries before you shop for ovens.

Step 1: Build a realistic job in Colab

Open a new Colab notebook. We'll make a small but typical ETL-plus-scoring job. It reads a CSV, cleans each row in Python, does some number crunching and writes the result out.

import time, numpy as np, pandas as pd

# Make an input file: 1 million orders, a few tens of MB
rng = np.random.default_rng(0)
n = 1_000_000
pd.DataFrame({
    "user_id": rng.integers(0, 50_000, n),
    "amount":  rng.normal(50, 20, n).round(2),
    "country": rng.choice(["GB", "US", "DE", "FR", "IN"], n),
}).to_csv("orders.csv", index=False)

RATES = {"GB": 1.0, "US": 0.79, "DE": 0.85, "FR": 0.85, "IN": 0.0095}

def load():                        # I/O: disk read + CSV parsing
    return pd.read_csv("orders.csv")

def clean(df):                     # Python called once per row
    df["gbp"] = df.apply(lambda r: r.amount * RATES[r.country], axis=1)
    return df

def compute(df):                   # dense numeric work
    X = rng.standard_normal((50_000, 256))
    W = rng.standard_normal((256, 256))
    for _ in range(20):            # stand-in for a scoring model
        X = np.tanh(X @ W / 16)
    return df.groupby("user_id")["gbp"].sum(), X

def save(df):                      # I/O: formatting + disk write
    df.to_csv("out.csv", index=False)

Step 2: Time each stage on the wall clock

time.perf_counter() is a high-resolution, monotonic clock. It counts sleeping and waiting too, so it measures wall-clock time, which is the time your users actually wait. Only the difference between two readings means anything.

timings = {}
def timed(name, fn, *args):
    t0 = time.perf_counter()
    out = fn(*args)
    timings[name] = time.perf_counter() - t0
    return out

df = timed("load", load)
df = timed("clean", clean, df)
_  = timed("compute", compute, df)
timed("save", save, df)

total = sum(timings.values())
for k, v in timings.items():
    print(f"{k:8s} {v:6.2f}s  {v/total:5.1%}")

Here is an example output. Your numbers will vary with the Colab machine you get:

load       0.60s   3.5%
clean     12.80s  75.7%
compute    1.30s   7.7%
save       2.20s  13.0%

Gotcha: run the job twice. The first run pays one-off costs such as imports and a cold file cache. Benchmark the second run.

Tip: time.process_time() counts only CPU time and leaves out waiting. If wall time is much bigger than process time, the job is waiting on something, usually I/O.

Step 3: Ask cProfile why

The stage timings tell you which stage is slow. cProfile tells you which functions inside it are slow.

import cProfile, pstats

with cProfile.Profile() as pr:
    clean(load())
pstats.Stats(pr).sort_stats("cumulative").print_stats(8)

Read three columns. ncalls is how many times a function ran. tottime is the time spent inside the function itself. cumtime is that time plus everything the function called. Sort by cumulative to find the expensive branch of the call tree, and by tottime to find hot loops. Here the red flag is a ncalls of 1,000,000 for the lambda, with pandas row-access functions called millions of times under it. That is interpreter overhead. Little of it is arithmetic.

Gotcha: cProfile adds a cost to every function call, so code made of many tiny calls looks slower under the profiler than it really is. Take totals from perf_counter and use cProfile only to see where the time goes.

(From Python 3.15 the same profiler can also be imported as profiling.tracing. import cProfile keeps working.)

Step 4: Sort the time into compute, I/O and overhead

An accelerator speeds up the compute bucket. It doesn't make the disk faster, and it doesn't run your Python lambda faster.

Step 5: Amdahl's law, the ceiling on any speed-up

Gene Amdahl made this argument in 1967. If a fraction p of the run time can be sped up by a factor s, the rest still takes as long as it did:

def amdahl(p, s):
    return 1 / ((1 - p) + p / s)

p = 0.077                       # share of time in compute
print(amdahl(p, 20))            # 20x faster chip    -> 1.08x overall
print(amdahl(p, float("inf")))  # infinitely fast    -> 1.08x overall

Even a chip that did the compute in zero time would save just 1.3 of 16.9 seconds. The ceiling is 1 / (1 - p). If 90% of the job can be sped up, the most you can get is 10x. At 50% it's 2x. That ceiling also ignores the cost of moving data to the accelerator and back, which makes the real number worse.

The honest fix here is no chip at all. Replacing the per-row apply with a vectorised column operation removes most of the overhead bucket. The job gets faster, and as a side effect a bigger share of what's left is compute. That's when an accelerator starts to be worth discussing.

The seven-part workload profile

For the rest of the course, every processor is judged against seven questions about the workload:

1. Parallelism: how much of the work is independent?
2. Arithmetic intensity: how much maths per byte moved?
3. Memory bandwidth versus compute: which one runs out first?
4. Branching: is the control flow uniform or full of ifs?
5. Data size and transfer: how big is the data, and how often does it move?
6. Batch or latency: throughput over many items, or one fast answer?
7. Precision: does it need FP64, or will lower precision do?

Running this elsewhere: on Azure or AWS the method is the same. Time the stages with perf_counter on a normal VM before you ask for accelerator quota. If compute is a small share of the total, no instance type will save you.

Key takeaways

• Measure before you choose: use perf_counter for wall-clock totals and cProfile to see where the time goes.
• Sort the run time into compute, I/O and overhead. Accelerators only help the compute bucket.
• Amdahl's ceiling is 1 / (1 - p). With 8% compute, even an infinitely fast chip gives about 1.08x.
• Python per-row loops are overhead. Vectorise them first.
• The workload decides, then the processor. The seven-part profile is how we'll describe workloads from now on.