Lesson 2: Resource Hierarchy and Multi-Project Observability

Why the hierarchy is the first thing you design

Every Google Cloud resource sits in a tree. The organization node is the root, tied to your Cloud Identity or Google Workspace domain. Folders are optional groups beneath it, and projects hold the actual resources: VMs, GKE clusters, buckets. Every resource except the root has exactly one parent.

The tree matters because policy flows down it. An IAM role granted on a folder is inherited by every folder and project inside it, and organization policies cascade the same way. Think of a building's master-key system. A key cut for "Floor 3" opens every room on floor 3, and you can't take that access away by changing the lock on one room.

The IAM rule behind this is short: a resource's effective allow policy is the union of its own allow policy and everything it inherits from its ancestors. A child can add access. It cannot remove access granted higher up with an allow policy. Deny policies can, but that comes later. So you design the tree so that grants made high up are ones you're happy for everything below to inherit.

Designing the layout: a worked example

Take Acme Retail. It has two teams (Payments and Storefront), each with an app, plus a platform team that runs shared services. One common, exam-friendly layout is application-centric, with one project per environment:

acme.com (organization)
├── fldr-shared            # platform-owned: logging, monitoring, networking
│   ├── acme-logging       # central log buckets
│   └── acme-monitoring    # scoping project for the metrics scope
├── fldr-payments          # Payments team gets roles/editor on its dev only
│   ├── payments-dev
│   ├── payments-staging
│   └── payments-prod
└── fldr-storefront
    ├── storefront-dev
    ├── storefront-staging
    └── storefront-prod

Why one project per environment? A project is a strong boundary for IAM, quotas, billing and blast radius. A developer with broad rights in payments-dev gets nothing in payments-prod. If dev and prod share a project, you're stuck with fiddly per-resource IAM.

An alternative is environment-first: fldr-prod and fldr-nonprod at the top, with apps underneath. That makes "lock down all of production" a single organization policy on one folder. The trade-off is that team-wide grants then have to be repeated in each environment folder. Pick the axis you most often want to apply policy along, and nest the other one beneath it. Folders can nest up to 10 levels deep, and one parent can hold at most 300 direct child folders.

Gotcha: Granting roles/owner on the organization "just for setup" gives that person owner on every project, now and in the future. Inheritance works in your favour only when broad grants stay low in the tree.

Multi-project metrics: the metrics scope

Cloud Monitoring reads metrics through a metrics scope. Every project has one, and the project that owns it is the scoping project. Scoping projects and metrics scopes map one to one. When you add other projects as monitored projects, one set of dashboards and alerts can see all of them. By default a metrics scope can include 375 projects. You can ask for a quota increase, but Google warns that very large scopes can slow queries down.

Best practice is a dedicated scoping project (acme-monitoring) that runs no workloads of its own. On-call engineers get Monitoring roles there and never need access to the app projects.

# Add payments-prod to the metrics scope owned by acme-monitoring.
# Note the beta track and the projects/ prefix on the positional argument.
gcloud beta monitoring metrics-scopes create projects/payments-prod \
    --project=acme-monitoring

The command fails if a project doesn't exist, if you lack permission on either project, or if the project is already in the scope.

The same thing in Terraform, which is how you'd keep it repeatable:

resource "google_monitoring_monitored_project" "payments_prod" {
  # Format: locations/global/metricsScopes/<SCOPING_PROJECT_ID_OR_NUMBER>
  metrics_scope = "locations/global/metricsScopes/acme-monitoring"
  name          = "payments-prod"   # the project being monitored
}

Multi-project logs: aggregated sinks

Metrics scopes give you a view across projects. Logs work differently: you route copies of them. An aggregated sink created on a folder or the organization with --include-children routes matching logs from that node and every descendant project, including projects created later.

Step 1: create a central log bucket:

gcloud logging buckets create central-logs \
    --project=acme-logging --location=global --retention-days=365

Step 2: create the sink at the organization level:

gcloud logging sinks create org-to-central \
  logging.googleapis.com/projects/acme-logging/locations/global/buckets/central-logs \
  --organization=123456789012 \
  --include-children \
  --log-filter='severity>=WARNING OR logName:"cloudaudit.googleapis.com"'
# Use --folder=FOLDER_ID instead of --organization to cover only one folder.

Step 3: grant the sink's writer identity (a Google-managed service account) permission on the destination. Without this grant, the sink is created but no logs arrive.

WRITER=$(gcloud logging sinks describe org-to-central \
  --organization=123456789012 --format='value(writerIdentity)')

gcloud projects add-iam-policy-binding acme-logging \
  --member="$WRITER" --role=roles/logging.bucketWriter

The Terraform equivalent is google_logging_organization_sink (or google_logging_folder_sink) with include_children = true.

Trade-off: --include-children is non-intercepting. Logs still flow to each child project's own sinks as well, so you may pay to store the same log twice. --intercept-children creates an intercepting sink instead: it takes over routing for the node and all its children, so sinks in the child projects no longer get those logs. That is powerful, but it can quietly break a team's own exports. Also note there is a limit of 200 sinks per folder or organization.

What the exam asks about this

Expect scenarios such as: "Ops needs one dashboard across 40 projects with no access to app projects" (answer: a dedicated scoping project with the 40 projects added to its metrics scope). Or: "Security needs all audit logs, including from future projects, kept centrally" (answer: an organization-level aggregated sink with --include-children). Distractors often suggest creating a sink in every project, which misses new projects. Another common distractor tries to restrict an inherited grant by changing a child's allow policy.

Key takeaways

• Organization → folders → projects; IAM and organization policies inherit downward, and the effective allow policy is a union.
• Use one project per environment for isolation; choose application-first or environment-first folders based on where you most often apply policy.
• Metrics scope = one scoping project plus monitored projects (375 by default). Add them with gcloud beta monitoring metrics-scopes create or google_monitoring_monitored_project.
• Aggregated sink = a sink at folder or organization level with --include-children, sent to a central log bucket.
• Always grant the sink's writer identity roles/logging.bucketWriter on the destination project.