Right Chip, Right Job: CPUs, GPUs, TPUs and Beyond

What's inside

1. Level 1: The Workload Decides, Not the ChipFree
   Learn why an accelerator is not an automatic speed-up: measure where a real Python job spends its time, then use Amdahl's law to work out the most any processor could gain.
2. Level 2: Parallelism: How Much Work Is IndependentFree
   Tell data parallelism from task parallelism and dependent work, and find out how many independent operations a job really offers to wide hardware.
3. Level 3: Arithmetic Intensity: FLOPs per Byte
   Work out arithmetic intensity, the number of floating-point operations done for each byte moved from memory, for common kernels by counting both in code.
4. Level 4: Memory Bandwidth Versus Compute
   Measure the memory bandwidth and the peak compute of the machine in front of you, and use the two numbers to decide whether a kernel is limited by memory or by compute.
5. Level 5: Branching and Memory Access Patterns
   See how data-dependent branches and irregular, pointer-chasing memory access cut real throughput far below what the bandwidth and FLOP figures suggest.
6. Level 6: Precision and Data Size
   Understand how the choice of number format (FP64, FP32, BF16, FP16, INT8) changes the bytes moved, the size of the working set and the hardware throughput you can reach, and what it costs in accuracy.
7. Level 7: Transfer, Batching and Latency
   Account for where data lives and what it costs to move it, and tell throughput-oriented batch work apart from latency-bound single requests.
8. Level 8: The Roofline Model in Practice
   Combine peak compute, memory bandwidth and arithmetic intensity into a roofline plot, place real Python jobs on it and read off what limits each one.
9. Level 9: The CPU as a Latency Machine
   Read a modern CPU core against the roofline: out-of-order execution, branch prediction, deep cache hierarchies and why it is good at irregular, branchy and serial code.
10. Level 10: Multicore, Threads and NUMA
    Scale CPU work across cores, and learn how the GIL, thread-pool settings, memory bandwidth limits and NUMA placement cap the speed-up.
11. Level 11: SIMD on x86: SSE, AVX2 and AVX-512
    Understand how x86 vector instructions process many values per instruction and how NumPy and compiled code use them without you writing intrinsics.
12. Level 12: SIMD on Arm: NEON and SVE
    Learn Arm's fixed-width NEON and its vector-length-agnostic SVE and SVE2, and what that means for code that must run well on different Arm server cores.
13. Level 13: Arm Versus x86 for Server Workloads
    Weigh moving production code from x86 to Arm cloud CPUs, looking at price-performance, memory bandwidth per core, software compatibility and porting risk.
14. Level 14: CPU Matrix Units: AMX and SME
    Find out how in-core matrix engines such as Intel AMX and Arm SME speed up low-precision matrix maths on the CPU, and when they let a CPU take on work you would otherwise send to an accelerator.
15. Level 15: GPU Execution Model: SIMT and Warps
    Understand how a GPU runs thousands of threads in lockstep groups (warps or wavefronts), hides memory latency by keeping many threads in flight, and loses speed when threads in a group branch differently.
16. Level 16: GPU Memory Hierarchy and Coalescing
    Learn the GPU memory hierarchy, from registers and shared memory through L2 cache to HBM or GDDR, and how memory access patterns decide whether a kernel gets close to peak bandwidth.
17. Level 17: Tensor Cores and Low-Precision GPU Math
    Understand the matrix units inside modern GPUs, the low-precision formats they speed up, and why the advertised peak TFLOPS figure seldom matches your code.
18. Level 18: NVIDIA: CUDA Stack and Product Landscape
    Map NVIDIA's data-centre GPU generations, interconnects and CUDA software stack as of late 2026, with every claim dated, to see why the ecosystem matters as much as the silicon.
19. Level 19: AMD: Instinct GPUs and ROCm
    Assess AMD's Instinct accelerators and the ROCm and HIP software stack, including their memory-capacity strengths and where software maturity still differs from CUDA.
20. Level 20: Intel GPUs, oneAPI and SYCL
    Review Intel's GPU line-up and its oneAPI and SYCL programming model as a vendor-neutral route, with an honest, dated account of product roadmap changes and how mature the ecosystem is.
21. Level 21: Inference Specialists I: Groq's LPU and SRAM Dataflow
    Learn how a deterministic, compiler-scheduled chip that keeps model weights in on-chip SRAM changes the roofline for token-by-token LLM decode, and what the late-2025 NVIDIA–Groq licensing deal means for a developer choosing it in 2026.
22. Level 22: Inference Specialists II: Cerebras, SambaNova and Peers
    Compare wafer-scale (Cerebras) and reconfigurable dataflow (SambaNova RDU) designs against the roofline, and treat vendor tokens-per-second claims as dated, conditional evidence rather than settled fact.
23. Level 23: NPUs: Neural Engines on Phones and Laptops
    Understand what the neural processing units in Apple, Qualcomm, Intel and AMD client chips do well (low-power, quantised, fixed-shape inference) and why they are reached through vendor runtimes, not general-purpose code.
24. Level 24: FPGAs: Building the Circuit Around the Workload
    Learn how FPGAs trade peak throughput for custom pipelines with deterministic latency, and why the development cost (HLS, synthesis times, scarce skills) is usually the deciding factor.
25. Level 25: DPUs and SmartNICs: Offloading the Data Centre Tax
    Understand how data processing units such as NVIDIA BlueField, AMD Pensando and AWS Nitro take networking, storage and security work off the host CPU, and when that matters to application developers.
26. Level 26: The Processor Map: Every Chip on One Roofline
    Bring CPUs, GPUs, TPUs, cloud ASICs, inference specialists, NPUs, FPGAs and DPUs together on one dated comparison of compute, bandwidth, memory capacity, programmability and availability.
27. Level 27: The Naive Port: Paying the PCIe Toll
    Move a NumPy job to the GPU with CuPy in Colab, watch it get slower, and measure how host-to-device transfers over PCIe can cost more than the compute they feed.
28. Level 28: Small Batches and Kernel Launch Overhead
    See why many tiny GPU operations lose to the CPU, because launch latency and an under-filled device dominate, and how batching and kernel fusion restore the advantage.
29. Level 29: Branchy Code and Python Loops on the GPU
    Learn why data-dependent branching causes warp divergence, and why a Python loop that issues one GPU call per element is the slowest code you can write.
30. Level 30: Missing GPU Paths and Device Memory Limits
    Deal with the two blunt walls of a port: libraries or functions with no GPU implementation, which force silent round trips to the CPU, and datasets larger than device memory.
31. Level 31: Precision and Determinism: Same Code, Different Answers
    Understand why GPU results can differ from CPU results and from run to run: weak FP64 throughput on many GPUs, TF32 and reduced-precision defaults, and non-deterministic parallel reductions.
32. Level 32: Stop Guessing: Profiling GPU Code Properly
    Replace wall-clock guesses with real GPU profiles, using the PyTorch profiler and NVIDIA Nsight Systems to see transfers, launches, idle gaps and synchronisation stalls on a timeline.
33. Level 33: Case Study, Part 1: A pandas and NumPy Job Moved Naively
    Take a realistic ETL-plus-numerics pipeline (pandas joins, groupby, a row-wise apply and a NumPy scoring step) and swap in cuDF and CuPy naively, recording every way it disappoints.
34. Level 34: Profile on the CPU First: Finding the Real Hot Spot
    Before touching an accelerator, use cProfile, py-spy and line_profiler to find where the case-study pipeline actually spends its time, and use Amdahl's law to work out the most any port could gain.
35. Level 35: Vectorise First: Winning on the CPU You Already Have
    Remove Python-level loops and row-wise applies with NumPy, pandas vectorised operations and Numba, often getting most of the speed-up before any GPU is involved.
36. Level 36: Isolating the Hot Kernel Behind a Clean Boundary
    Refactor the pipeline so the compute-heavy kernel sits behind one function with explicit array inputs and outputs, data stays resident on the device, and a backend can be swapped without touching the rest of the code.
37. Level 37: Choosing a Layer: PyTorch versus JAX and XLA
    Compare PyTorch's eager-first model (with torch.compile) against JAX's trace-and-compile model through XLA, including static shapes, recompilation and portability to TPUs.
38. Level 38: Portable Runtimes: ONNX Runtime and SYCL
    Learn when to export to ONNX and let ONNX Runtime execution providers target CPU, GPU or NPU, and when SYCL/oneAPI offers single-source C++ portability across vendors.
39. Level 39: CUDA versus ROCm: What Really Ports to AMD
    Understand how ROCm and HIP mirror CUDA, what HIPIFY translates automatically, and where custom kernels, libraries and tooling still need work, with every support claim dated to its source.
40. Level 40: Case Study, Part 2: The Proper Port, Measured
    Rebuild the level-33 pipeline using profile, vectorise, isolate and choose-a-layer, then compare before and after with honest end-to-end measurements, and note how the same job would run on Azure and AWS.
41. Level 41: GPU Dataframes: pandas and Polars on the GPU
    Learn when a dataframe job speeds up on a GPU by running the same pandas and Polars pipeline on the CPU, under RAPIDS cuDF's pandas accelerator mode and under Polars' GPU engine in Colab, then reading which operations moved to the GPU and which quietly fell back to the CPU.
42. Level 42: GPU SQL and String-Heavy Queries
    See how scans, filters, joins, aggregations and string operations such as regex, tokenising and parsing behave on GPU SQL and dataframe engines, and why string columns with very different lengths can reduce the gain.
43. Level 43: Graph Analytics on GPUs
    Speed up graph algorithms such as PageRank, connected components and shortest paths with GPU graph libraries, for example cuGraph through the NetworkX backend, and see why irregular graph traversal still works well on a GPU when the graph is large enough.
44. Level 44: Monte Carlo Simulation on Accelerators
    Move a NumPy Monte Carlo pricing or risk simulation to CuPy and JAX, handling random number generation, the reduction step and reproducibility correctly, so that millions of independent paths actually use the parallel hardware.
45. Level 45: Vector Search on GPUs
    Build and query approximate nearest-neighbour indexes on CPU and GPU with FAISS and cuVS, and learn how the type of index, batch size and recall target decide whether the GPU is worth paying for.
46. Level 46: TPUs for Non-ML Linear Algebra
    Use JAX on a Colab TPU for scientific linear algebra that is not machine learning, such as large matrix products, solvers and simulations, and learn where the TPU's matrix units and default precision help or hurt.
47. Level 47: FPGAs for Low-Latency Streams
    Understand why FPGAs are chosen for deterministic, microsecond-scale stream processing such as market data, packet handling and sensor pipelines, and what high-level synthesis and vendor toolchains really require from a software team.
48. Level 48: NPUs for On-Device Inference
    Run a small model on a laptop or phone NPU through ONNX Runtime execution providers or vendor runtimes, and learn what operator support, quantisation and power limits mean for on-device features.
49. Level 49: Single-Request Latency Traps
    Measure why an accelerator that wins on throughput can lose on the latency of a single request, once kernel launch overhead, data transfer and the wait to fill a batch are added to every request.
50. Level 50: Tiny Models and Small Workloads
    Learn why small models, such as gradient-boosted trees, small MLPs and classic ML, often run fastest on a well-tuned CPU, and how to recognise a workload that is too small for an accelerator.
51. Level 51: Irregular Memory Access and Pointer Chasing
    See how scattered gathers, hash lookups, sparse structures and pointer-chasing code break memory coalescing and caching on accelerators, and which restructurings, such as sorting, bucketing or changing data layout, get the performance back.
52. Level 52: When I/O Is the Real Bottleneck
    Profile pipelines where disk, network, decompression or deserialisation take most of the time, so a faster processor changes almost nothing, and learn the data-loading fixes that come before any hardware change.
53. Level 53: Licences, Quota and Availability
    Account for the constraints outside the code: per-core and per-socket software licensing, cloud accelerator quota and capacity, regional availability, and reservation commitments that can block or change a hardware decision.
54. Level 54: Quantisation for Inference
    Apply post-training quantisation (INT8, FP8, 4-bit weight formats) to a model, measure the effect on accuracy, memory and speed, and understand which processors actually have fast paths for each format.
55. Level 55: Batching and Serving Engines
    Compare static, dynamic and continuous batching, and serving engines such as vLLM, TensorRT-LLM, SGLang, llama.cpp and ONNX Runtime, to see how KV-cache management and scheduling decide throughput and latency.
56. Level 56: Cost per Token and When a CPU Wins
    Build a cost-per-token and cost-per-request model from measured throughput, utilisation and current instance or API pricing, and identify the traffic patterns where CPU inference or a hosted inference specialist is cheapest.
57. Level 57: A Repeatable Benchmark Method
    Design a benchmark that holds up under review: a representative workload, warm-up, synchronisation, repeated runs with variance reported, fixed versions, and correctness checks before any speed claim.
58. Level 58: Normalising Results: Cost, Energy and Effort
    Turn raw benchmark numbers into a fair comparison across processors by normalising for cost, energy, utilisation and the engineering effort needed to port and maintain each option.
59. Level 59: The Processor Selection Checklist
    Put the course together as a step-by-step selection checklist: workload profile, roofline position, software ecosystem, limits and traps, inference economics, and availability, applied to a real candidate workload.
60. Level 60: TPUs in Depth: Systolic Arrays, XLA and Pods
    How a Google TPU actually executes a program, read against the workload profile and the roofline model: the piece the processor lessons left out, so the selection checklist can be applied to TPUs with confidence.
61. Level 61: Final Challenge: Three Workloads, Three Choices Exam
    Review the whole course by choosing processors for three contrasting workloads, backing each choice with a profile, roofline reasoning, benchmark evidence, cost figures and a written rationale that covers trade-offs and risks.

Access

The first 2 levels are free with a free account. Every level, the podcast edition and the AI tutor come with All Access at £4.99/month or any Creator plan — see pricing.

Start free →