Lesson 1: The Google Cloud Data Services Map

Why start with a map

When someone asks you to "put this on Google Cloud", the first mistake is usually not picking the wrong service. It is not knowing a service exists, or searching the docs under a name Google stopped using six months ago. In April 2026 Google renamed several data products in a single month. The commands, APIs and IAM roles kept their old names. So your Terraform says dataproc while the console says Managed Service for Apache Spark. This level gives you one page that sorts every service into a group and lists each one's old and new names.

Think of it like a kitchen. Some places store food (warehouses), some cook it (processing), some move it between stations (pipelines and messaging), one keeps the inventory list (catalogue), and one puts the dish on the plate (BI). Every service has one main station, even if it can help out at another.

The one-page map

Two notes on placement. BigQuery appears twice on purpose. Its managed storage is the warehouse, and its engine also queries data it does not store. Airflow sits under pipelines because it orchestrates the jobs that move data. It does not process the data itself. That difference matters later, when you will be tempted to run heavy transforms inside Airflow workers.

The 2026 renames, with dates

• 3 April 2026: Dataproc on Compute Engine and Serverless for Apache Spark were merged under one brand, Managed Service for Apache Spark. You still choose between clusters and serverless. That choice now lives inside one product.
• 10 April 2026: Dataplex Universal Catalog became Knowledge Catalog. Discovery, lineage, data quality scans and business glossaries carried on without any migration.
• 15 April 2026: Cloud Composer became Managed Service for Apache Airflow. The docs now call the generations Legacy Gen 1, Gen 2 and Gen 3, where you may remember Composer 1, 2 and 3.
• April 2026: Looker Studio went back to being Data Studio. Reports, data sources and sharing kept working. Looker, the enterprise platform, kept its name.

The pattern is that Google moved towards plain names built around open source (Spark, Airflow) and stopped putting "Looker" on two very different products.

Why the CLI didn't change: a walkthrough

A product name is marketing. API endpoints, gcloud command groups and IAM role IDs are contracts. Millions of scripts, Terraform states and org policies depend on them. Renaming roles/dataproc.editor would break every IAM binding that grants it. So Google renamed the brands and froze the identifiers.

Here is what you actually type in October 2026:

# Managed Service for Apache Spark: still the dataproc command group and API
gcloud dataproc clusters list --region=europe-west2
gcloud dataproc batches list --region=europe-west2   # serverless Spark batches

# Managed Service for Apache Airflow: still composer
gcloud composer environments list --locations=europe-west2

# Knowledge Catalog: still dataplex
gcloud dataplex entries search "orders" --project=my-proj --location=global

# Which data APIs are enabled? The service names show the old brands
gcloud services list --enabled --filter="name:(dataproc OR composer OR dataplex)"

Realistic output from the last command:

NAME                        TITLE
composer.googleapis.com     Cloud Composer API
dataplex.googleapis.com     Cloud Dataplex API
dataproc.googleapis.com     Cloud Dataproc API

IAM looks the same: Knowledge Catalog still uses roles/dataplex.catalogAdmin and permissions such as dataplex.entries.get. In Terraform you keep writing google_dataproc_cluster, google_composer_environment and the google_dataplex_* resources.

Gotcha: if you search the console or docs for "Dataproc" and get a page titled Managed Service for Apache Spark, you are in the right place. Going the other way, a code review that "fixes" dataproc to spark breaks the build. Keep a rename table in your team wiki. Once you keep a table, the renames stop causing trouble.

Retired and retiring: what replaces what

Both dates are in the past. If you take over a project, check what it is still running:

# Show each environment's image version; anything composer-1.* or composer-2.0.* is past end of life
gcloud composer environments list --locations=europe-west2 \
  --format="table(name, config.softwareConfig.imageVersion)"

NAME          IMAGE_VERSION
etl-prod      composer-2.0.32-airflow-2.2.5     # past end of life: plan a move to Gen 3
reports-dev   composer-3-airflow-2.10.5         # fine

Best practice: treat a former name in a design document as a warning sign. A design that says "Data Catalog tags" or "Composer 1" was written for a platform that no longer exists in that form.

Key takeaways

• Seven groups: warehouses and lakehouses (BigQuery, BigLake), query engines (BigQuery's engine with BigLake and Omni), integration and pipelines (Dataflow, Datastream, Data Fusion, Airflow), streaming and messaging (Pub/Sub, Managed Kafka), big data processing (Managed Service for Apache Spark), catalogues and governance (Knowledge Catalog), BI (Looker, Data Studio).
• April 2026 renames: Dataproc became Managed Service for Apache Spark, Cloud Composer became Managed Service for Apache Airflow, Dataplex Universal Catalog became Knowledge Catalog, and Looker Studio became Data Studio again.
• The API, gcloud and IAM identifiers (dataproc, composer, dataplex) are contracts and were left unchanged so existing automation keeps working.
• Data Catalog shut down on 30 January 2026, and its replacement is Knowledge Catalog. Composer 1 and 2.0.x reached end of life on 15 September 2026, and you should move to Gen 3 or to 2.1.x or later.