Lesson 2: The Lakehouse and the Data Intelligence Platform

Why this level matters

The Databricks Intelligence Platform section is only 6% of the exam, about 2 or 3 of the 45 scored questions. The rest of the course builds on it, though. Auto Loader, Lakeflow Jobs and Unity Catalog privileges are easier to follow once you know where things run and where data sits. A reminder from Level 1: the exam guide belongs to Databricks and is free on its certification page. This course is independent of Databricks and can't promise you a pass.

The problem the lakehouse solves

For years most companies ran two systems side by side:

Running both means copying data from the lake into the warehouse. You end up with two copies, two sets of permissions, and arguments about which number is correct.

A data lakehouse combines the benefits of both. It keeps data in cheap object storage, in open formats, and adds warehouse features on top: ACID transactions, schema enforcement, governance and fast queries. BI, data engineering and ML then work on one copy of the data, which serves as the single source of truth.

Analogy: a data lake is a huge warehouse floor with boxes stacked anywhere. A lakehouse uses the same cheap floor, adds a stock ledger that records every box moved, and puts a security desk at the door.

In that analogy, Delta Lake is the ledger and Unity Catalog is the security desk.

The Data Intelligence Platform

Databricks calls its product the Data Intelligence Platform. It is a lakehouse with AI added on top that learns your data's meaning (your table names, your jargon) to power features like natural-language queries. For this exam, keep the architecture underneath it in mind. It has three parts.

1. The control plane, run by Databricks

These are the backend services that Databricks manages in its own account. They include the web application (the UI you log into), job scheduling and orchestration, and workspace and cluster configuration. You never manage these servers yourself.

2. The compute plane, where your data is processed

The compute plane comes in two variants:

• Classic compute plane: the cluster VMs run in your own cloud account, inside your workspace's virtual network. You pay your cloud provider for the VMs.
• Serverless compute plane: the compute runs in Databricks' account, in the same region as your workspace. Network boundaries isolate each customer's workspace.

Level 4 covers compute types and cost in depth. For now, remember one rule: the control plane manages and the compute plane processes.

3. Cloud object storage, where your data lives

Table data is stored as files in cloud object storage. In a classic setup that is your own storage bucket, and you can connect serverless compute to it too. Serverless workspaces can also use Databricks-managed default storage. Either way, your tables are files in object storage. They are not stored inside the control plane.

           ┌──────────── CONTROL PLANE (Databricks account) ─────────────┐
  You ───▶ │ Web UI · job scheduler · workspace and cluster config        │
           └──────────────────────────────┬──────────────────────────────┘
                                          │ starts and instructs
           ┌──────────── COMPUTE PLANE ───▼──────────────────────────────┐
           │ Classic: VMs in YOUR cloud account                          │
           │ Serverless: compute in Databricks' account (your region)    │
           └──────────────────────────────┬──────────────────────────────┘
                                          │ reads and writes files
           ┌──────────── CLOUD OBJECT STORAGE ───────────▼───────────────┐
           │ Parquet data files + _delta_log/  (your tables)             │
           └─────────────────────────────────────────────────────────────┘

Walkthrough: see the lakehouse in one table

Run this in a notebook or the SQL editor. Navigating the workspace comes in Level 3, so here we only look at the result.

-- Three-level name: catalog.schema.table. Unity Catalog owns this namespace.
CREATE TABLE main.demo.orders (order_id INT, amount DOUBLE);

INSERT INTO main.demo.orders VALUES (1, 19.99), (2, 5.00);

-- Ask Delta Lake about the table's physical layout
DESCRIBE DETAIL main.demo.orders;

Trimmed output:

format | location                                   | numFiles
delta  | s3://my-bucket/.../tables/8f3c...           | 1

Three things to notice:

1. format = delta. You didn't ask for Delta. Delta Lake is the default format on Databricks, so any table you create without naming a format is a Delta table.
2. location is a path in cloud object storage. That is where the data actually lives.
3. If you listed that folder, you would see Parquet data files plus a _delta_log/ directory:

part-00000-....snappy.parquet   <- the data, in open Parquet format
_delta_log/00000000000000000000.json   <- commit 0: CREATE TABLE
_delta_log/00000000000000000001.json   <- commit 1: INSERT

The transaction log is what turns a plain folder of Parquet files into a table with ACID transactions. A write only counts once its commit file lands, so a reader never sees half a job.

Gotcha: don't delete or edit files under a table's path by hand. The log and the files will no longer match. Always change data with SQL or DataFrame commands.

Delta Lake and Unity Catalog in one sentence each

• Delta Lake is the open-source storage layer that adds a transaction log to Parquet files. It gives you ACID transactions and schema enforcement on cheap object storage.
• Unity Catalog is the unified, fine-grained governance layer for data and AI on Databricks. It controls who can access what, organises objects as catalog.schema.table, and tracks lineage.

Rule of thumb: Delta Lake makes the data reliable, and Unity Catalog makes it governed and discoverable. Later levels cover each in depth.

What the exam asks about this

Expect short scenario questions such as: "Which component hosts the web application?" (the control plane). "Where do classic cluster VMs run?" (your cloud account). "Why choose a lakehouse over a separate lake and warehouse?" (one copy of the data in open formats, with ACID and governance). Watch for distractors that say your table data is stored in the control plane, or that swap the jobs of Delta Lake and Unity Catalog.

Key takeaways

• A lakehouse combines cheap, open-format object storage with warehouse features (ACID, schema enforcement, governance), so all workloads share one copy of the data.
• The control plane (Databricks' account) holds the web app and management services.
• The compute plane processes data. Classic compute runs in your cloud account and serverless compute runs in Databricks' account.
• Table data lives as files in cloud object storage, never in the control plane.
• Delta Lake is the default table format: Parquet files plus a _delta_log transaction log.
• Unity Catalog governs access, the catalog.schema.table namespace, and lineage.