Wide hardware needs independent work
In Level 1 we said the workload decides, not the chip. This level covers the first thing to check about any workload: how much of the work can happen at the same moment?
Think of a supermarket. Opening 40 tills only helps if 40 customers are ready to pay. If one customer has to finish before the next can start (say each one needs the receipt of the person ahead), you get no gain from the other 39 tills. A GPU is a supermarket with tens of thousands of tills. A CPU core has a handful of very fast ones.
Here is the scale, from NVIDIA's published figures:
- Tesla T4 (the usual free Colab GPU): 40 SMs, each holding up to 1,024 resident threads, so about 40,960 threads in flight.
- H100 SXM5: 132 SMs × 2,048 resident threads each, so about 270,336 threads in flight.
The GPU hides latency by switching between those threads. If your job gives it 3,000 independent pieces of work, most of the chip sits idle and you pay for all of it anyway.
Three kinds of work
- Data-parallel: the same operation on many items, and no item depends on another.
y = x * 2 + 1over a million elements gives you a million independent operations. - Task-parallel: different jobs that don't depend on each other, such as resizing images while you parse logs, or training five models with different settings.
- Serial (dependent): step i needs the result of step i−1. This is called a loop-carried dependency. Extra lanes cannot help, because the next step's input does not exist yet.
Walkthrough: sort these loops
import numpy as np
x = np.random.rand(1_000_000)
# A: data-parallel. Each output uses only its own input.
y = np.sqrt(x) * 2.0 + 1.0 # 1,000,000 independent ops
# B: reduction. Many inputs collapse to one value.
total = x.sum()
# C: scan (prefix sum). Output i depends on outputs 0..i-1.
running = np.cumsum(x)
# D: non-linear loop-carried dependency. Truly serial.
def logistic_path(r, x0, n):
out = np.empty(n)
v = x0
for i in range(n):
v = r * v * (1.0 - v) # needs the previous v; can't be split
out[i] = v
return out
A fits a GPU naturally. D does not: a GPU gains nothing on a single path. Lateral thinking can still help, though. If you need 100,000 paths with different r values, the paths are independent of each other, and you have 100,000-way data parallelism again. B and C fall in between, and they need a closer look.
Reductions and scans: dependent, but rescuable
A naive sum (acc += x[i]) looks serial. But addition is associative, so you can regroup it into a tree: add pairs, then pairs of pairs, and so on.
import math
n = 1_000_000
print(math.ceil(math.log2(n))) # 20 -> 20 tree levels instead of 999,999 sequential adds
The first level has 500,000 independent adds, the next 250,000, and so on. Parallelism shrinks at each level, so a reduction never keeps the hardware as busy as case A does.
Scans (cumsum, running max, running balance) can also be parallelised with a tree-shaped parallel prefix algorithm. These do more total work than the serial loop and need several passes, so expect a smaller gain than with a plain element-wise operation. A linear recurrence such as an exponential moving average, ema[i] = a*x[i] + (1-a)*ema[i-1], can be rewritten as a scan. A non-linear one like case D cannot.
Gotcha: regrouping changes floating-point results. The NumPy docs warn that
x.cumsum()[-1]may not equalx.sum(), becausesumuses pairwise summation. Their example: forb = np.array([1, 2e-9, 3e-9] * 1000000),b.cumsum()[-1]gives1000000.0050045159andb.sum()gives1000000.0050000029. Any parallel reduction reorders the additions, so don't expect bit-identical output from CPU and GPU.
Count your independent operations
For each step of a job, count how many independent operations it exposes per call, not per job.
T4_THREADS = 40 * 1024 # 40,960
def verdict(name, independent_ops):
ratio = independent_ops / T4_THREADS
print(f"{name:28s} {independent_ops:>10,} {ratio:6.2f}x T4 capacity")
verdict("element-wise on 1M floats", 1_000_000)
verdict("batch of 32 x 512 features", 32 * 512)
verdict("per-row loop, 3 values/call", 3)
element-wise on 1M floats 1,000,000 24.41x T4 capacity
batch of 32 x 512 features 16,384 0.40x T4 capacity
per-row loop, 3 values/call 3 0.00x T4 capacity
The last line is the classic disappointing port. A Python loop over DataFrame rows sends the GPU three numbers at a time, so the chip is idle almost all the time no matter how big the whole table is.
Amdahl's law in numbers
The serial part of a job caps the speed-up you can get:
def amdahl(parallel_fraction, lanes):
return 1 / ((1 - parallel_fraction) + parallel_fraction / lanes)
print(round(amdahl(0.90, 40_960), 2)) # 10.0 -> 10% serial caps you near 10x
print(round(amdahl(0.95, 40_960), 2)) # 19.99
print(round(amdahl(0.99, 40_960), 2)) # 99.76
Best practice: before you rent a big chip, find the serial fraction. Moving from 90% to 99% parallel is worth more than any hardware upgrade.
Try it in Colab
Run the snippets above in a CPU runtime. Time np.sqrt(x) against logistic_path at the same n with %timeit. Then vectorise D across 100,000 r values (an array of v, one loop over time steps) and watch the per-step work jump from 1 to 100,000.
On Azure and AWS: the same counting applies before you pick any GPU VM. A job that exposes only a few thousand independent operations per call will under-use a cloud GPU just as it does a T4, and you pay by the hour either way.
Key takeaways
- Sort every hot loop into data-parallel, task-parallel or serial before you think about hardware.
- A T4 holds about 41k resident threads and an H100 SXM5 about 270k. Count your independent operations per call against those figures.
- Reductions and scans can be parallelised as trees, but with less parallelism and different floating-point rounding.
- Non-linear loop-carried dependencies are serial. Look for independence across paths, users or batches instead.
- Amdahl's law: a 10% serial fraction caps you near 10× no matter how many lanes you have.