Why a profile comes before a product
Level 1 gave you the map: every Google Cloud data service in its group. A map does not tell you where to go, though. To get that you need a description of the journey, and in data work that description is the analytics profile: ten fields that describe a workload without naming any product.
Think of it as a doctor's intake form. A good doctor does not prescribe anything until they know your symptoms, history and allergies. Engineers who open with "let's use Kafka" are prescribing first. The profile makes you describe the workload first. Most of the time two or three answers settle the choice, and you can drop half the map in five minutes.
The ten fields:
- Sources and volume: where the data comes from, how much arrives per day, how much is kept
- Batch or streaming: does data arrive in files or jobs, or as a constant flow of events?
- Freshness: the longest delay the business can accept between an event and an insight
- Query patterns and concurrency: dashboards, ad hoc SQL or point lookups, and how many run at the same time at peak
- Users: analysts (SQL and BI), engineers (pipelines, code) or scientists (notebooks, Python, Spark)
- Open formats: must the data stay in Iceberg, Delta Lake or Parquet so other engines can read it?
- Governance and lineage: classification, column-level access, audit, "where did this number come from?"
- Residency: which regions the data may be stored or processed in
- Team skills: what the team can run at 3 a.m. today, which is not the same as what it could learn
- Cost model: pay per use, a committed budget, or a hard monthly ceiling
Walkthrough: profiling a real workload
Here is a sample: a UK online retailer that wants live order monitoring and daily sales reporting. Write the profile as a file in the repo, next to the code it will justify.
# analytics-profile.yaml: one file per workload, reviewed like code
workload: retail-orders-monitoring
owner: data-platform-team
date: 2026-10-29 # profiles go stale; date them
sources:
- name: web-checkout-events # JSON events from the shop backend
volume_per_day: 40 GB # ~25M events/day, peak 1,200 events/s on Black Friday
- name: erp-postgres # nightly CDC from the order database
volume_per_day: 5 GB
retention: 3 years # total ≈ 50 TB raw; state the number, not "a lot"
arrival: streaming + batch # events stream; ERP lands nightly
freshness:
fraud_alerts: 5 s # business: "before the parcel is picked"
sales_dashboard: 15 min
finance_reports: 24 h
queries:
pattern: [dashboards, ad-hoc SQL]
peak_concurrency: 60 # 9:00 Monday: 60 dashboard users at once
point_lookups: none # no per-customer API reads
users: {analysts: 25, engineers: 4, scientists: 2}
open_formats: "none required" # ask! a single 'Iceberg only' here changes everything
governance: [PII columns masked for analysts, lineage for finance KPIs]
residency: UK/EU only # contract clause, see legal ticket LEG-212
team_skills: [SQL strong, Python ok, no Kafka/Spark ops experience]
cost_model: predictable monthly budget, ≤ £8k
Notice what the file does. Each vague word ("real-time", "big", "secure") has become a number or a rule you can check. "Real-time" turned into three freshness targets, and only one of them is in seconds. That split is the most useful thing a profile gives you.
Gotcha: always record peak concurrency and volume. An average of 50 events per second tells you nothing about Black Friday.
Knock-out answers: fields that rule services in or out
Some answers act as filters, not preferences. Apply them before any scoring:
| Profile answer | Immediate effect |
|---|---|
| Sub-second freshness or per-key reads in milliseconds | A batch load or a scheduled query can't do it. You need a streaming path that ends in a serving store such as Bigtable, which offers single-digit-millisecond reads and writes for single rows |
| Seconds-level alerts on events that land in the warehouse | Consider a streaming pipeline, or BigQuery continuous queries, which run SQL all the time and can send results to Pub/Sub, Bigtable or Spanner |
| "Iceberg only", so other engines must read the same copy | Prefer formats you control. BigLake tables for Apache Iceberg in BigQuery are managed by BigQuery but keep the data in your Cloud Storage buckets |
| Data already in Delta Lake | BigQuery can read it through BigLake external tables for Delta Lake. Writes stay with your Delta engine |
| Strict residency | Choose the location at creation time. A BigQuery dataset's location cannot be changed later; you can only copy or recreate it. Pub/Sub topics take a message storage policy that lists the allowed regions |
| No ops skills for clusters | Managed and serverless options score higher than anything you have to patch and size yourself |
Run the retail profile through these filters. The 5-second fraud alert is the strict field, so streaming ingestion is required. The 15-minute and 24-hour needs are easy. Residency fixes the region before anything is built. The lack of Kafka experience counts against running a self-managed broker. You haven't picked a product yet, but the list is already shorter.
Trade-off: each knock-out narrows your options, so make the business justify it. A "sub-second" demand often turns out to be "faster than yesterday's report". Ask: what decision changes if the data is 5 minutes old rather than 5 seconds?
Who answers which field
Engineers fill in profiles badly when they guess the business fields. Business owners fill them in badly when they guess the technical ones. Split the form between them:
FIELD ANSWERED BY HOW TO GET IT
freshness business "What breaks if it's an hour late?"
users & concurrency business + eng headcount from business; peak from query logs
residency business/legal contracts, regulation, not engineering opinion
governance (what/why) business/compliance which data is sensitive, who may see it
cost model business/finance budget shape: variable vs committed
sources & volume engineers measure: storage metrics, row counts, event rates
batch vs streaming engineers how sources actually emit data today
query patterns engineers sample real queries, don't imagine them
open formats engineers (+ arch) which other engines must read the data
team skills engineering lead honest inventory, not the CV
Best practice: if a business field is blank, mark it
UNKNOWN - owner: <name>. Don't fill it with a guess. An open question with an owner gets answered. A guessed value turns into a requirement nobody agreed to.
Key takeaways
- The profile describes the workload, never the product. Choose products only after it's complete.
- Turn every vague word into a number: GB/day, peak events per second, freshness in seconds, minutes or hours, peak concurrent users.
- Split freshness by use case. Usually only one consumer needs seconds.
- Knock-out fields (sub-second reads, Iceberg-only, residency, missing ops skills) filter services before you score anything. Residency is especially costly to get wrong because a BigQuery dataset's location is fixed when it is created.
- Business owners answer freshness, residency, governance intent and cost model. Engineers measure volume, arrival, query patterns, formats and skills.
- Date the profile and keep it in version control. It will justify your selection record later.