Lesson 2: Cloud IAM and Organization Policies for Data

Who can touch the data?

Every pipeline you build has two kinds of identity behind it. One is people: analysts, engineers and auditors. The other is machines: Dataflow workers, Dataproc VMs, Composer schedulers. Cloud IAM decides what each identity may do. Organization policies set the outer limits that no IAM grant can get past. Section 1.1 of Google's exam guide lists "Identity and Access Management (e.g., Cloud IAM and organization policies)" first among the security and compliance topics, and many scenario questions depend on it.

Think of a building. IAM works like key cards: a key card says who opens which door. Organization policies work like the building code: no tenant may install a gas heater, whatever keys they hold.

IAM in one sentence, then the details

An IAM allow policy binds a principal (user:, group:, serviceAccount:) to a role (a bundle of permissions) on a resource. Resources form a hierarchy: organization → folder → project → resource. For BigQuery, the resources below the project are the dataset and then the table or view. A grant made higher up is inherited by everything below it, and allow policies are additive. So you cannot take away at table level something you granted at project level.

Three kinds of role

Exam reflex: if one answer gives someone roles/editor "to make it work", it is almost always a distractor. Look for the narrowest predefined role granted at the lowest sensible level.

Walkthrough: an analyst who can query one dataset

Say Priya needs to read sales_mart in project acme-analytics, but nothing else. BigQuery splits this into two permissions: running a query job (granted on a project) and reading the data (granted on a dataset or table).

# 1. Let Priya run query jobs in the project. jobUser gives no data access by itself.
gcloud projects add-iam-policy-binding acme-analytics \
  --member="group:sales-analysts@acme.com" \
  --role="roles/bigquery.jobUser"
# Grant to a group, not to Priya personally: when people join or leave, only group membership changes.

-- 2. Let the group read ONE dataset (BigQuery calls a dataset a SCHEMA in SQL).
GRANT `roles/bigquery.dataViewer`
ON SCHEMA `acme-analytics`.sales_mart
TO "group:sales-analysts@acme.com";

-- 3. Even narrower: a dashboard service account that reads one table only.
GRANT `roles/bigquery.dataViewer`
ON TABLE `acme-analytics.sales_mart.daily_revenue`
TO "serviceAccount:looker-studio@acme-analytics.iam.gserviceaccount.com";

-- Take a grant back with REVOKE ... ON ... FROM ...

The GRANT statement accepts the resource types SCHEMA, TABLE, VIEW and EXTERNAL TABLE. Project and folder grants are made outside SQL, with gcloud or the console.

Gotcha: granting dataViewer without jobUser gives an error the first time the user runs a query. Granting jobUser without dataViewer gives "Access Denied" on the table. You need both.

Organization policies: guardrails across many projects

IAM answers "who may do X?". An organization policy answers "may anyone do X here at all?". You set a constraint on the organization, a folder or a project, and every project below inherits it.

Two constraints come up again and again in data scenarios:

• constraints/gcp.resourceLocations restricts where new resources such as buckets, datasets and clusters can be created. It applies only to new resources; ones that already exist are not moved.
• constraints/iam.managed.disableServiceAccountKeyCreation stops anyone from downloading long-lived JSON keys for service accounts. Older material names the legacy version, iam.disableServiceAccountKeyCreation. Organizations created on or after 3 May 2024 have it switched on by default, together with iam.automaticIamGrantsForDefaultServiceAccounts. That second constraint stops default service accounts from getting Editor automatically.

# eu-only.yaml: enforce on the "EU data" folder; every project below it inherits the rule
name: folders/123456789/policies/gcp.resourceLocations
spec:
  rules:
  - values:
      allowedValues:
      - in:eu-locations      # a value group covering EU regions and multi-regions

gcloud org-policies set-policy eu-only.yaml
# Now `bq mk --location=US ...` in any project under that folder is rejected.

Exam reflex: when a question says "across all projects", "prevent", or "even if a developer has Owner", the answer is an organization policy, not more IAM grants.

Dedicated service accounts for pipelines

If you set nothing, Dataflow workers and Dataproc VMs run as the Compute Engine default service account (PROJECT_NUMBER-compute@developer.gserviceaccount.com). In older projects that account often still has Editor. Any job, and any bug in a job, then has broad access to the whole project. Instead, give each pipeline its own identity:

# 1. Create a service account for one pipeline
gcloud iam service-accounts create orders-dataflow \
  --display-name="Orders streaming pipeline"
SA=orders-dataflow@acme-data.iam.gserviceaccount.com

# 2. Grant only what it needs
gcloud projects add-iam-policy-binding acme-data \
  --member="serviceAccount:$SA" --role="roles/dataflow.worker"   # needed to run work items
gcloud pubsub subscriptions add-iam-policy-binding orders-sub \
  --member="serviceAccount:$SA" --role="roles/pubsub.subscriber" # its source only
# ...plus roles/bigquery.dataEditor on the target dataset only

# 3. Let the launching engineer or CI "act as" it (roles/iam.serviceAccountUser
#    gives iam.serviceAccounts.actAs), then launch with it:
python orders_pipeline.py --runner=DataflowRunner \
  --service_account_email=$SA ...

The same pattern works for the other two services:

• Dataproc: gcloud dataproc clusters create etl --service-account=$SA. The account needs roles/dataproc.worker plus access to the buckets the jobs read and write.
• Cloud Composer: gcloud composer environments create ... --service-account=$SA. The account needs roles/composer.worker. Add other permissions only when your DAGs need them.

Trade-off: one account per pipeline means more accounts to manage. In return you get a small blast radius and a clear audit trail: logs show which pipeline touched a table. Because key creation is disabled, nobody ever handles a key file. Workloads on Google Cloud get their credentials from the service account attached to them.

What the exam asks about this

Expect scenarios such as "analysts must query dataset A but not B" (dataset-level dataViewer plus project-level jobUser), "stop all teams creating resources outside the EU" (gcp.resourceLocations at org or folder level), and "a Dataflow job runs with excessive permissions" (a dedicated service account with roles/dataflow.worker and narrow data roles). Wrong answers usually grant basic roles, grant at too high a level, or try to enforce an org-wide rule one project at a time.

Key takeaways

• Prefer predefined roles, granted to groups, at the lowest level that works. Use custom roles only when predefined ones are too broad.
• BigQuery access usually needs a project-level job role (jobUser) plus data roles on a dataset or table (dataViewer/dataEditor). GRANT ... ON SCHEMA/TABLE sets the data roles in SQL.
• Grants are inherited downwards and are additive. You cannot take access away lower down.
• Organization policies are guardrails that IAM cannot override. Know gcp.resourceLocations and iam.managed.disableServiceAccountKeyCreation.
• Never run pipelines as the Compute Engine default account. Give Dataflow, Dataproc and Composer their own service accounts with dataflow.worker, dataproc.worker or composer.worker plus narrow data access.