HPC on GCP and Azure: Test, Benchmark, Decide

What's inside

1. Level 1: The Project: One Benchmark From Laptop to DecisionFree
   Learn why choosing HPC compute on GCP or Azure has to be settled by testing, and set up the single project that carries one benchmark from your laptop to a written, evidence-backed recommendation.
2. Level 2: Loosely vs Tightly Coupled WorkloadsFree
   Learn to classify a workload by how much its parallel parts talk to each other, and see why that one answer decides whether you need a low-latency fabric at all.
3. Level 3: Finding the Bottleneck: Compute, Memory, I/O, GPU
   Learn to tell whether a workload is limited by compute, memory bandwidth, storage I/O or the GPU, using simple measurements instead of guesswork.
4. Level 4: Defining Measurable Success Criteria
   Turn vague goals into measurable success criteria (time to solution, scaling efficiency, time to start, cost per result) with thresholds and weights agreed before any test runs.
5. Level 5: Laptop Toolchain: Compilers, MPI and mpirun
   Install a compiler and an MPI library on your laptop, build OSU Micro-Benchmarks and STREAM from source, and learn the basics of launching MPI ranks with mpirun.
6. Level 6: Measuring Memory Bandwidth With STREAM
   Run STREAM properly, with an array big enough to beat the cache and threads pinned to cores, and read its Copy, Scale, Add and Triad results.
7. Level 7: OSU Baselines and a Reproducible Results Log
   Run OSU latency, bandwidth and allreduce tests over shared memory on your laptop and save every result with enough metadata that anyone can reproduce it later.
8. Level 8: Containers From First Principles
   Learn what a container really is (namespaces, cgroups and a layered filesystem sharing the host kernel) and why that matters for HPC, where the kernel and drivers stay on the host.
9. Level 9: Your First Benchmark Image: STREAM in a Dockerfile
   Write a Dockerfile that builds and runs STREAM, push it to a local registry, and refer to it by an immutable digest so every later test uses exactly the same bits.
10. Level 10: Multi-Stage Builds for Lean HPC Images
    Use multi-stage builds to compile OSU in a full toolchain stage and ship only the binaries and runtime libraries in a slim final image.
11. Level 11: MPI Inside Containers
    Learn how MPI behaves when it is packaged in an image: choosing an MPI implementation, ABI compatibility, PMIx, and the hybrid model where the launcher on the host must match the MPI in the container.
12. Level 12: libfabric and UCX: Choosing the Fabric Path
    Learn how MPI reaches the network through libfabric or UCX providers, and how to check and control which transport (shm, tcp, verbs) a run actually uses.
13. Level 13: Host Drivers and the Kernel Boundary
    Learn why GPU and RDMA drivers stay on the host, how user-space libraries in the image must match the host's kernel modules, and how device plugins and container toolkits bridge the gap.
14. Level 14: /dev/shm, Memlock and Other Container Limits
    Fix the container defaults that quietly break HPC runs: a small /dev/shm, memlock ulimits that block RDMA memory registration, and CPU limits that hurt pinning.
15. Level 15: Apptainer vs OCI Images
    Compare Apptainer's single-file SIF images and user-level execution with OCI images, and decide which format each target platform (Slurm, Kubernetes, batch services) will need.
16. Level 16: Kubernetes for Batch: A Local Cluster
    Learn the Kubernetes pieces a batch user actually needs (API server, scheduler, nodes, kubectl and declarative YAML) by building a multi-node local cluster with kind.
17. Level 17: Pods: Running One Benchmark Task
    Run STREAM as a single Pod and learn how resource requests and limits, QoS classes, restart policy and logs shape the behaviour of a batch task.
18. Level 18: Jobs and Indexed Jobs for Parameter Sweeps
    Use Jobs to run to completion and retry on failure, and Indexed Jobs to fan out a sweep where each Pod picks its own task from JOB_COMPLETION_INDEX.
19. Level 19: Persistent Storage: PVs, PVCs and StorageClasses
    Persist inputs and results beyond a Pod's life with PersistentVolumeClaims, and learn how StorageClasses and access modes (RWO, RWX) map onto the shared filesystems you will test in the cloud.
20. Level 20: Hand-Wiring Multi-Pod MPI
    Wire an MPI run across several Pods by hand with a headless Service, stable hostnames and a hostfile, so you feel exactly the gaps that Kueue, JobSet and the MPI Operator fill next.
21. Level 21: Kueue: Queues, Quotas and Admission Locally
    Install Kueue on your local kind cluster and learn how ResourceFlavors, ClusterQueues and LocalQueues decide when a batch Job may start, so jobs wait in a queue instead of crowding the cluster.
22. Level 22: JobSet: Gang-Shaped Jobs with Several Roles
    Use JobSet to describe one run made of several coordinated Jobs, such as a launcher and a set of workers with stable hostnames, and to set what happens when one part fails.
23. Level 23: The MPI Operator: Running OSU Across Pods
    Install the Kubeflow MPI Operator and run the OSU point-to-point and collective benchmarks as an MPIJob, learning how the launcher reaches workers and generates the hostfile for you.
24. Level 24: Local Dress Rehearsal and What Local Cannot Tell You
    Combine Kueue, the MPI Operator and a PVC into one repeatable local benchmark run that stores its results, then compare it with your laptop baseline and list what only the cloud can measure.
25. Level 25: Terraform Core: Providers, Resources, Plan, Apply
    Learn the Terraform workflow from scratch with the google and azurerm providers: declare resources, read a plan carefully, apply it and destroy it, on resources that cost nothing.
26. Level 26: Terraform State: Remote Backends, Locking and Drift
    Understand why state is the most sensitive file in Terraform, how to keep it in a locked remote backend on GCS or Azure Storage, and how to find and handle drift.
27. Level 27: Modules, Variables and Reading the Platform's Code
    Structure Terraform as versioned modules with inputs and outputs, and learn to read and use your platform team's modules instead of writing your own.
28. Level 28: The Terraform Boundary: Platforms, Not Jobs
    Recognise that Terraform builds clusters, queues and storage but must not submit benchmark jobs, and choose how jobs will actually run on each platform you test.
29. Level 29: Harness Fundamentals: Pipelines, Connectors, Delegates
    Learn the Harness building blocks that you will use every day — organisations, projects, pipelines, stages, steps, connectors and delegates — and where each one runs.
30. Level 30: Federated Credentials: Keyless Access to GCP and Azure
    Configure Harness connectors that use OIDC with GCP Workload Identity Federation and Azure workload identity federation, so pipelines never hold long-lived cloud keys.
31. Level 31: Plan, Approval, Apply: Proving the Path Early
    Build a Harness pipeline that runs Terraform Plan, waits for a human Approval and then Applies, and prove it end to end with a trivial, free resource before trusting it with anything that costs money.
32. Level 32: Backstage Catalog: Reading catalog-info.yaml
    Learn how Backstage models software and infrastructure as catalog entities, and use the catalog to find the owners, pipelines and templates that your benchmark project depends on.
33. Level 33: Consuming Software Templates Before Authoring
    Run the platform team's existing Backstage Software Templates to scaffold repositories and environments, and learn to read a template so you know what it will do before you fill it in.
34. Level 34: End to End: Template to Pipeline to Cloud
    Join Backstage, Harness and Terraform into one flow, in which a template creates a project and its pipeline then plans, approves and applies a trivial resource and tears it down.
35. Level 35: Private Clusters and Running Jobs Through Harness
    Reach private GKE and AKS control planes from a delegate inside the network, and use Harness steps to apply job manifests and collect results, following the job-submission decision from Level 28.
36. Level 36: Authoring a Benchmark Template and the Handover Line
    Write your own Software Template for a benchmark environment on top of the platform team's modules, and agree clearly where their templates take over from yours.
37. Level 37: GKE Standard vs Autopilot for HPC Benchmarks
    Create GKE clusters through your delivery path and compare Standard and Autopilot for HPC, looking at node control, privileged needs, shared memory, placement and how each is billed — with every claim dated and checked against current Google Cloud documentation.
38. Level 38: Node Pools from Zero and Custom ComputeClasses
    Configure node pools that scale down to zero when idle, and use GKE custom ComputeClasses to state machine-family preferences and fallbacks, so the benchmark only pays for nodes while a job runs.
39. Level 39: Compact Placement: Running the MPIJob on GKE
    Run your MPI Operator OSU benchmark across GKE nodes, with and without a compact placement policy, to measure how much physical node proximity changes latency and collective performance.
40. Level 40: Spot and Dynamic Workload Scheduler with Kueue
    Get capacity in two different ways — Spot VMs for cheap capacity that can be taken back, and Dynamic Workload Scheduler flex-start with queued provisioning through Kueue for all-at-once capacity — and measure time to start and cost for each.
41. Level 41: Kueue on GKE: Queues, Quotas and Fair Sharing
    Bring the Kueue setup you tested locally onto GKE so that ClusterQueues, ResourceFlavors and admission checks control when the benchmark Jobs and JobSets start on node pools that scale from zero.
42. Level 42: Cloud Batch: Containers Without Running a Cluster
    Run the same benchmark image on Google Cloud Batch, which creates VMs for each job and deletes them afterwards, and see what you gain and lose compared with GKE.
43. Level 43: H4D and Cloud RDMA: Tightly Coupled MPI on GCP
    Run the tightly coupled OSU latency and bandwidth tests on H4D VMs over Cloud RDMA, and learn which parts of the image and host stack have to change for RDMA to work.
44. Level 44: Slurm Through Cluster Toolkit
    Deploy a small Slurm cluster with Google Cluster Toolkit blueprints and run the benchmark as a batch script, so you have the traditional HPC scheduler as a baseline.
45. Level 45: Slurm on GKE: The Slinky Operator
    Run Slurm inside Kubernetes with the Slurm Operator add-on for GKE, built on SchedMD's open-source Slinky project, and test whether one cluster can serve both Slurm users and Kubernetes batch jobs.
46. Level 46: GCP Storage I: Managed Lustre and Filestore
    Attach Google Cloud Managed Lustre and Filestore to your jobs through CSI-backed PVCs and measure how each holds up under parallel benchmark I/O.
47. Level 47: GCP Storage II: GCS FUSE and Staging Data
    Use Cloud Storage FUSE through the GKE CSI driver for input and output data, and decide when to stage data onto a file system and when to stream it from object storage.
48. Level 48: AKS for HPC Batch: The Same Manifests on Azure
    Run the Kubernetes benchmark on AKS with node pools that scale from zero, Spot node pools and Kueue, and record what had to change from GKE.
49. Level 49: Azure Batch: Pools, Tasks and Containers
    Run the benchmark image on Azure Batch pools and compare its pool-based model with Cloud Batch, where each job gets its own VMs.
50. Level 50: CycleCloud Workspace for Slurm
    Deploy Azure CycleCloud Workspace for Slurm, a free Marketplace solution template, and work out how a Marketplace deployment fits a delivery path that allows only Terraform.
51. Level 51: HB and HX VMs with InfiniBand
    Run the tightly coupled OSU tests on Azure HB- or HX-series VMs over InfiniBand, and document how the InfiniBand stack differs from Cloud RDMA on GCP.
52. Level 52: Azure Storage: Managed Lustre and NetApp Files
    Attach Azure Managed Lustre and Azure NetApp Files to AKS and Slurm jobs, then repeat the I/O tests you ran on GCP.
53. Level 53: Quotas, Capacity and Spot Checkpointing
    Plan for the two things that most often stop HPC runs in the cloud (quota you lack and capacity that is not there) and make Spot runs survive preemption with checkpoints.
54. Level 54: Licence Servers, Data Movement and Egress
    Deal with what sits around the compute: reaching commercial licence servers from short-lived nodes, moving data sets in and results out, and the egress charges that come with it.
55. Level 55: Security for HPC on Shared Platforms
    Secure the benchmark platform end to end: workload identity, private clusters, image provenance, the limits on privileged containers that RDMA access needs, and secrets that never touch a pipeline log.
56. Level 56: Cost Guardrails and Monitoring
    Put hard limits and visibility around every test: budgets, labels, automatic clean-up, and the metrics and logs that make benchmark runs comparable and catch waste.
57. Level 57: Fair Benchmarking: Normalise and Disclose
    Turn the raw results from both clouds into a fair comparison: same image wherever possible, controlled variables, repeated runs, and an honest note wherever the RDMA stack forced a difference.
58. Level 58: Scaling Efficiency, Time to Start, Cost per Result
    Calculate the three numbers the decision rests on: how well each option scales, how long a job takes from submission to first useful work, and what one finished result really costs.
59. Level 59: Weighted Scoring Without a Favourite
    Score every tested option against weighted criteria taken from your level 1-7 success definition, set the weights before looking at the totals, and test whether the ranking holds up.
60. Level 60: Final Challenge: The One-Page Recommendation Exam
    Bring the whole course together by writing a one-page, evidence-backed recommendation that takes your benchmark from laptop baseline to a defended choice of compute and storage on GCP or Azure.

Access

The first 2 levels are free with a free account. Every level, the podcast edition and the AI tutor come with All Access at £4.99/month or any Creator plan — see pricing.

Start free →