As a Specialist Solutions Architect at Databricks, I often hear the same questions from customers who are migrating dashboards from legacy BI tools to Databricks AI/BI Dashboards:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

In the first blog post in this series, I focused on the first two questions and showed how to recreate:

• context filters using parameters in dataset SQL, and

• “Only Relevant Values” filters using field filters and query-based parameters.

In the second post, I focused on cross-filtering and drill-through interactions.

This third post tackles the remaining question: “How do we do user-based filtering in AI/BI Dashboards?”

In Databricks, these controls live in Unity Catalog, not in the dashboard itself. AI/BI Dashboards query governed tables and views, and Unity Catalog enforces fine-grained access control before the data ever reaches the dashboard. I’ll walk through how to:

• Implement user-based filtering and RLS with dynamic views that use current_user() and is_account_group_member().

• Apply RLS directly on tables using row filters and protect sensitive fields with column masks.

• Scale these patterns across many tables and columns using ABAC tag policies and governed tags.

As in the previous posts, I’ll use the built-in samples.tpch dataset. I’ve also published the companion dashboard so you can import it into your workspace, follow along as you read, and adapt these patterns to your own Unity Catalog data.

1. How User-Based Filtering Maps to Unity Catalog

Before we dive into SQL, it’s useful to clarify where these responsibilities live in Databricks.

In many BI tools, user-specific security is often implemented close to the dashboard or semantic layer:

• You define user- or group-based rules that map principals to specific regions, customers, or business units.

• You may use identity-aware logic in filters or calculated fields.

• You may maintain a security table that drives which slice of data each user can see.

In Databricks, these controls live in Unity Catalog, not in AI/BI Dashboards:

• Object privileges on catalogs, schemas, tables, and views control whether a user can query a given object at all.

• Dynamic views, row filters, and column masks implement row-level security and masking at query time. They can inspect the current user and their groups and return different rows or values per user.

• AI/BI Dashboards simply query those governed tables and views. They never bypass Unity Catalog: any row filters or masks you define apply to every query, regardless of whether it comes from a notebook, Databricks SQL, or an AI/BI dashboard.

The result is conceptually similar to user-based filtering in traditional BI tools, but with one important shift: The security rules live with the data, not with a particular dashboard.

That’s especially important when:

• The same Unity Catalog tables power multiple AI/BI dashboards and external BI tools, and

• You embed AI/BI Dashboards into applications where thousands of users see the same dashboard definition, but each must see a different subset of data.

In the rest of this post, we’ll build up from that idea:

1. Use a dynamic view and a permission table to enforce RLS on a TPCH Sales dataset.

2. Show how to do similar things directly on tables with row filters and column masks.

3. Discuss how to scale those patterns across many tables and columns using ABAC tag policies and governed tags.

2. Building blocks: dynamic views, row filters, and column masks

To implement user-based filtering in Databricks, you really only need three Unity Catalog primitives: dynamic views, row filters, and column masks.

They all rely on the same core idea: At query time, Unity Catalog can look at who is running the query (and which groups they’re in), and then decide which rows and values to return.

2.1 Identity functions

The main functions you’ll use in policies are:

• current_user()
  Returns the current user’s identity (usually their email).

• is_account_group_member(’<group_name>’)
  Returns TRUE if the current user is a member of an account-level group.

You can call these functions from views and from SQL UDFs used by row filters and column masks.

2.2 Dynamic views

A dynamic view is just a normal SQL view whose logic depends on the current user or their groups.

You can:

• Filter rows based on current_user() or group membership.

• Mask or null out columns for certain users.

• Join to a separate permission table that maps users/groups to allowed regions, customers, etc.

Any AI/BI dataset that selects from a dynamic view automatically inherits its logic. You don’t need to add any special configuration in AI/BI itself.

We’ll use a dynamic view for our first TPCH Sales RLS example.

2.3 Row filters

A row filter attaches RLS logic directly to a table, instead of wrapping the table in a view:

• You define a SQL UDF that takes one or more columns as input and returns BOOLEAN.

• You attach it to a table with ALTER TABLE ... SET ROW FILTER ... ON (column[, ...]).

The row filter runs for every query and can call current_user() and is_account_group_member() internally. This is handy when you want:

• A stable table name (no extra view layer), or

• A single table that’s consumed by many tools, all of which should respect the same RLS.

We’ll look at both group-based and current_user() + permission-table examples later in the post.

2.4 Column masks

A column mask is similar, but operates at the column level:

• You define a SQL UDF that returns a “masked” value.

• You attach it with ALTER TABLE ... ALTER COLUMN ... SET MASK ....

This lets you:

• Show full values (for example, email or salary) only to certain groups.

• Show partially masked or null values to everyone else.

Think of it as the Unity Catalog side of “row-level security + column-level masking” that you might combine in legacy BI tools using data source filters and calculated fields.

Next, we’ll put these pieces together in a concrete example: enforcing region-based RLS on a TPCH Sales dataset using a combination of a base view, a permission table, and a dynamic view.

3. Implementing RLS with a dynamic view

Let’s start with a concrete scenario:

• You have a TPCH Sales dashboard shared across multiple sales teams.

• NA Sales Managers should see only AMERICA.

• EMEA Sales Managers should see only EUROPE.

• APAC Sales Managers should see only ASIA.

• Individual users may have their own custom regions.

In many traditional BI tools, you’d typically solve this with a user filter or data source filter that maps groups to Regions, and a security table to keep that mapping up to date.

In Databricks, we’ll use the same logical pattern – but move it into Unity Catalog:

1. Create a base view for TPCH Sales in a demo schema.

2. Create a permission table that maps principals (groups or users) to Regions.

3. Create a dynamic view that joins the base view to the permission table and applies row-level security based on current_user() and is_account_group_member().

4. Create an AI/BI dataset on top of that dynamic view and build a simple table visualization.

Throughout this section, we’ll:

• Read from samples.tpch (which everyone has).

• Create objects in main.demo_tpch (you can substitute another catalog/schema if needed).

All of the CREATE / INSERT statements in this section should be run in a notebook or SQL editor, not inside an AI/BI dataset. In AI/BI, you’ll just SELECT from the resulting view.

3.1 TPCH Sales base view

First, set up a simple demo schema and define a reusable base view for TPCH Sales:

-- Use the default UC catalog and create a demo schema
USE CATALOG main;
CREATE SCHEMA IF NOT EXISTS demo_tpch;

-- Base TPCH Sales view, reading from samples.tpch
CREATE OR REPLACE VIEW main.demo_tpch.tpch_sales_base AS
SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

3.2 Region access permission table

Next, create a permission table that describes who is allowed to see which Region.

-- Optional: you can keep security tables in the same schema
-- or create a separate one, e.g. main.demo_security
CREATE SCHEMA IF NOT EXISTS main.demo_tpch;

CREATE TABLE IF NOT EXISTS main.demo_tpch.tpch_region_access (
  principal_type STRING,   -- 'group' or 'user'
  principal      STRING,   -- group name or user email
  region         STRING    -- must match tpch_sales_base.region
);

INSERT INTO main.demo_tpch.tpch_region_access VALUES
  ('group', 'NA Sales Managers',        'AMERICA'),
  ('group', 'EMEA Sales Managers',      'EUROPE'),
  ('group', 'APAC Sales Managers',      'ASIA'),
  ('user',  'some.user@databricks.com', 'ASIA'),
  ('user',   current_user(),            'AFRICA');

What this does:

• The first three rows grant access based on account-level groups

• The fourth row grants access to ASIA to a specific user, even if they are not in one of those groups.

• The fifth row uses current_user() to grant you, the person running this SQL, access to AFRICA. When you execute the INSERT, Unity Catalog evaluates current_user() to your own email.

If you’re not in any of the NA/EMEA/APAC Sales Managers groups and you’re not some.user@databricks.com, the only applicable rule for you will be the one that says you can see AFRICA. We’ll see the effect of that in a moment when we query through the dynamic view.

3.3 Dynamic view with region-level RLS

Now create a dynamic view that applies row-level security by joining the base TPCH Sales view to the permission table and checking the current user’s identity and groups:

CREATE OR REPLACE VIEW main.demo_tpch.tpch_sales_rls AS
SELECT s.*
FROM   main.demo_tpch.tpch_sales_base AS s
WHERE EXISTS (
  SELECT 1
  FROM   main.demo_tpch.tpch_region_access a
  WHERE  a.region = s.region
    AND (
      (a.principal_type = 'group'
       AND is_account_group_member(a.principal))
      OR
      (a.principal_type = 'user'
       AND a.principal = current_user())
    )
);

This view enforces RLS as follows. For each row in tpch_sales_base, it looks for a matching rule in tpch_region_access based on region and either group or user principle. If no matching rule exists for the current user and that region, the row is filtered out.

You can verify this by running:

SELECT DISTINCT region
FROM main.demo_tpch.tpch_sales_rls;

If you are only granted access via the (’user’, current_user(), ‘AFRICA’) row, you should see: AFRICA.

From now on:

• Any query against main.demo_tpch.tpch_sales_rls returns only the regions granted to the current user.

• This applies uniformly whether the query comes from a notebook, Databricks SQL, or an AI/BI Dashboard.

• You can add or revoke access simply by inserting or deleting rows in main.demo_tpch.tpch_region_access – you don’t need to change the view logic.

3.4 Using the dynamic view in an AI/BI dataset

With main.demo_tpch.tpch_sales_rls in place, using it in AI/BI Dashboards is straightforward. You don’t need to re-implement any RLS logic in AI/BI – the dataset just selects from the governed view.

Create the TPCH Sales (RLS Dynamic View) dataset:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_rls;

Build a simple dashboard page to visualize the dataset. In the companion dashboard, I created a RLS with dynamic view page based on this dataset:

When you view this page, if your only grant is the row we inserted with (’user’, current_user(), ‘AFRICA’), the table will show only rows where the region is AFRICA.

The security logic lives in Unity Catalog (dynamic view + permission table). The dashboard just selects from tpch_sales_rls and automatically respects row-level security for each viewer.

4. Implementing RLS on tables with row filters and column masks

In the previous section, we implemented row-level security for TPCH Sales using a dynamic view and a permission table. That pattern works well when you want a named, shareable view to point AI/BI datasets at.

Unity Catalog also lets you attach RLS and masking logic directly to tables using:

• Row filters – control which rows a user can access in a table.

• Column masks – control what values they see in specific columns.

These policies are evaluated in Unity Catalog at query time and apply to all compute – SQL warehouses, notebooks, and AI/BI Dashboards. Unlike dynamic views, they keep the table name unchanged, which can be important when the same table is shared across many tools.

In this section, we’ll:

1. Create a TPCH Sales table in main.demo_tpch for row filters and masks.

2. Attach a group-based row filter that restricts Regions.

3. Attach a user-based row filter that uses current_user() and a permission table.

4. Add a column mask to protect a sensitive column.

Any AI/BI dataset that selects from this table will automatically respect these policies. You don’t need to configure anything special in AI/BI.

All of the statements below should be run in a notebook or SQL editor. AI/BI Dashboards just query the resulting table.

4.1 TPCH Sales table for filters and masks

Row filters and column masks only apply to tables (and a few other relation types), not to views. To keep things simple, we’ll materialize the tpch_sales_base view from Section 3 into a Delta table that we can attach policies to:

-- Use the same demo catalog and schema as before
USE CATALOG main;
USE SCHEMA demo_tpch;

-- Create a physical table from the base view for row filters and masks
CREATE OR REPLACE TABLE main.demo_tpch.tpch_sales_table AS
SELECT *
FROM main.demo_tpch.tpch_sales_base;

From now on, we’ll attach row filters and masks to main.demo_tpch.tpch_sales_table.
If you point an AI/BI dataset at this table instead of the dynamic view, the behavior will be controlled by these table-level policies.

4.2 Group-based row filter on Region

First, let’s attach a row filter that enforces the same Region rules we used in the dynamic view, but purely based on account-level groups:

-- Row filter function that decides which regions each group can see
CREATE OR REPLACE FUNCTION main.demo_tpch.tpch_region_filter(p_region STRING)
RETURNS BOOLEAN
RETURN
  CASE
    WHEN is_account_group_member('NA Sales Managers')   THEN p_region = 'AMERICA'
    WHEN is_account_group_member('EMEA Sales Managers') THEN p_region = 'EUROPE'
    WHEN is_account_group_member('APAC Sales Managers') THEN p_region = 'ASIA'
    ELSE FALSE
  END;

Attach it to the table:

ALTER TABLE main.demo_tpch.tpch_sales_table
  SET ROW FILTER main.demo_tpch.tpch_region_filter ON (region);

Effect:

• Whenever anyone queries main.demo_tpch.tpch_sales_table, Unity Catalog evaluates tpch_region_filter(region) for each row.

• If the user is in NA Sales Managers, only rows where the region is AMERICA are returned. If they’re in EMEA Sales Managers, they only see EUROPE; APAC Sales Managers see ASIA.

• Users not in any of these groups see no rows from this table.

If you point a dataset at:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_table;

Any visualizations based on such a dataset will now respect the group-based Region logic without going through the dynamic view.

4.3 User-based row filter with current_user() and a permission table

Group-based rules are great for broad roles, but you may need finer control – different users seeing different subsets of customers, accounts, or regions.

We can reuse the same pattern as in Section 3 – current_user() + a permission table – but this time embed it in a row filter function instead of a dynamic view.

Example: restrict access by customer_id:

-- Permission table mapping users to customers they can see
CREATE TABLE IF NOT EXISTS main.demo_tpch.customer_access (
  user_email  STRING,
  customer_id BIGINT
);

-- Example grants
INSERT INTO main.demo_tpch.customer_access VALUES
  ('some.user@databricks.com', 889),
  (current_user(),             1111);  -- Give yourself access

Now create a row filter function that consults this table. To avoid ambiguous name resolution with the customer_id column on the table, we’ll use a parameter name p_customer_id:

CREATE OR REPLACE FUNCTION main.demo_tpch.tpch_customer_filter(p_customer_id BIGINT)
RETURNS BOOLEAN
RETURN EXISTS (
  SELECT 1
  FROM   main.demo_tpch.customer_access a
  WHERE  a.user_email  = current_user()
    AND  a.customer_id = p_customer_id
);

Attach it to the same table. Because a table can have only one row filter, we’ll replace the Region filter from the previous subsection in this example:

ALTER TABLE main.demo_tpch.tpch_sales_table
  DROP ROW FILTER;

ALTER TABLE main.demo_tpch.tpch_sales_table
  SET ROW FILTER main.demo_tpch.tpch_customer_filter ON (customer_id);

Effect:

• Whenever someone queries main.demo_tpch.tpch_sales_table, Unity Catalog evaluates tpch_customer_filter(customer_id) for each row.

• For a given user, only rows whose customer_id appears in main.demo_tpch.customer_access
  for current_user() are returned.

• In the sample data above, you will only see orders for customer 1111, while some.user@databricks.com will see orders for customer 889.

You can verify this quickly:

SELECT DISTINCT customer_id
FROM main.demo_tpch.tpch_sales_table
ORDER BY customer_id
LIMIT 20;

If your only mapping is (current_user(), 1111), this query should return just 1111.

To use this in AI/BI Dashboards, you can point a dataset directly at the table:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_table;

In the companion dashboard, I created a RLS with row filter page based on this dataset:

When you open it, the table and visuals only show data for customers you are allowed to see according to customer_access, without any RLS logic in the dashboard itself.

4.4 Bonus: masking sensitive columns with a column mask

Row filters decide which rows a user can see. Sometimes you also need to partially hide sensitive values within those rows – for example, masking customer names, emails or phone numbers for most users while leaving them fully visible for a small group.

Unity Catalog column masks handle this at the column level using the same pattern: a SQL UDF that can branch on current_user() or group membership.

Suppose we want:

• Users in a PII Full Access group to see full customer names.

• Everyone else to see a partially masked version (for example, just the first few characters).

First, define a masking function:

CREATE OR REPLACE FUNCTION main.demo_tpch.mask_customer_name(name STRING)
RETURNS STRING
RETURN
  CASE
    WHEN is_account_group_member('PII Full Access') THEN name
    ELSE concat(substr(name, 1, 3), '***')
  END;

Attach it as a mask on the table:

ALTER TABLE main.demo_tpch.tpch_sales_table
  ALTER COLUMN customer_name
  SET MASK main.demo_tpch.mask_customer_name;

Effect:

• Users in the PII Full Access group see the full customer_name value.

• All other users see a masked version like Cus*** instead of Customer#000001111.

• The mask is enforced for every query against tpch_sales_table – notebooks, SQL editor, and AI/BI Dashboards – including exports.

Here is what it looks like in the companion dashboard:

5. Scaling RLS with ABAC and governed tags

Everything we’ve done so far (dynamic views, row filters, and column masks) is defined directly on individual Unity Catalog objects. That’s fine for a handful of tables, but it becomes hard to manage when you have dozens of catalogs, hundreds of schemas, and thousands of tables. This is exactly the problem that attribute-based access control (ABAC) with governed tags is designed to solve in Unity Catalog.

At a high level, ABAC adds three building blocks on top of the mechanisms we already used:

• Governed tags – account-level tags like sensitivity, business_domain, or region_scope, with a controlled set of allowed values. You attach these tags to catalogs, schemas, tables, or columns.

• Policy UDFs – reusable SQL UDFs that implement row-filter or column-mask logic, similar to the functions we wrote earlier, but intended to be reused across many datasets.

• ABAC policies – centrally managed policies that say “when a tagged object matches these conditions, apply this row filter or column mask for these principals.” Policies can be attached at the catalog, schema, or table level and inherit down the hierarchy.

Databricks recommends using ABAC as the primary way to apply row filters and column masks at scale, and reserving table-by-table configuration for special cases. Conceptually, you can think of ABAC as “lifting” the patterns from Section 4 into a central policy layer:

1. Tag the data once – apply governed tags to catalogs, schemas, tables, and columns that participate in RLS or masking.

2. Register reusable UDFs – define shared row-filter and mask functions in a governance schema (for example, governance.region_filter() and governance.mask_customer_name()).

3. Create ABAC policies ​​– define policies that attach those UDFs to tagged objects based on tag conditions and target groups.

4. Let tags drive behavior – as new tables and columns are tagged, the appropriate row filters and masks are applied automatically by Unity Catalog.

From an AI/BI Dashboards perspective, the experience is the same as in Sections 3 and 4: the dataset SQL stays simple, and the dashboard filters and visualizations work as usual. The difference is that the security logic is now centralized in ABAC policies and tags instead of being embedded directly into each table or view.

6. Summary and next steps

In many BI tools, user-based filtering and row-level security are often implemented close to the dashboard or semantic layer using user/group mappings, security tables, and identity-aware logic. In Databricks, the key shift is that these controls move into Unity Catalog, and AI/BI Dashboards simply query governed tables and views.

In this post, we walked through three main patterns:

• Dynamic views and permission tables (Section 3)
  We built main.demo_tpch.tpch_sales_rls on top of a base TPCH Sales view and a tpch_region_access table. The dynamic view uses current_user() and is_account_group_member() to return different Regions for different users and groups. AI/BI datasets that query this view automatically inherit the row-level security.

• Table-level row filters and column masks (Section 4)
  We materialized TPCH Sales into main.demo_tpch.tpch_sales_table and attached a row filter that looks up allowed customer_id values in customer_access based on current_user(). We also added a column mask for customer_name, showing full names only to a privileged group. Any dataset that selects from this table sees the combined effect of RLS and masking without any extra logic in the dashboard.

• ABAC and governed tags (Section 5)
  We then zoomed out to show how ABAC can apply the same kinds of row filters and masks at scale, using governed tags, reusable policy UDFs, and central ABAC policies. Instead of configuring each table or view by hand, you tag data once and let policies attach the right filters and masks automatically.

Across all three patterns, the core idea is the same: AI/BI Dashboards stay simple; Unity Catalog enforces who sees which rows and what values.

The companion dashboard brings these ideas together in a concrete, runnable example. If you import it into your workspace and wire it up to the views and tables from this post, you can see exactly how the visuals behave for different users as Unity Catalog applies dynamic views, row filters, and column masks behind the scenes.

Combined with the first two posts in this series (Part 1 and Part 2), you now have a practical set of patterns for implementing filtering, drill-through, and row-level security in Databricks AI/BI Dashboards on top of Unity Catalog.

• The Problem: A “Full Refresh” on Kafka-fed pipelines can cause permanent data loss if older records have aged out of the topic (TTL).

• The Strategy: Use a “Backup-and-Rebase” workflow: archive existing data, identify the last processed offsets, and point the new pipeline to that exact starting position.

• The Execution: This guide demonstrates how to manually configure startingOffsets in Spark to bridge the gap between historical backups and new Kafka data.

• The Result: Seamless streaming table updates with zero data loss or record duplication.

The Constraint: Kafka TTL & Streaming Table Immutability

In Spark Declarative Pipelines (SDP), situations may arise when you need to alter a bronze Streaming Table that is being fed by Apache Kafka. This can pose a challenge as you cannot manually alter Streaming Tables via Alter Table commands.

This is further complicated by the fact that Kafka topics are configured with a finite retention period (Time-to-live or TTL), meaning older records eventually age out. Since a pipeline Full Refresh clears the target Delta Table, you cannot simply update your pipeline definition and full refresh from the source, as older records will be missing. The diagram below illustrates this situation. The Kafka cluster contains user-4 who has already been ingested, and users 5 and 6 who still need to be ingested into our Lakehouse. The older user records (1, 2, and 3) have aged out of the topic’s TTL.

As new records are constantly being appended to this topic, how can I update my pipeline and alter my result table without missing new records (users 5, 6, etc), dropping old records (users 1, 2, and 3), or duplicating records (user-4)? The following example has been fabricated to show the solution. The actual reason for implementing this will vary by use-case.

Example: Ingesting JSON via Kafka

Imagine a pipeline is ingesting JSON data from Kafka. This JSON data contains three high-level fields: name, country, and email. It also contains two nested fields “event” and “device” which contain information about what actions users are taking from which devices.

{”user”:”user-2”,”email”:”user-2@example.com”,
“country”:”CA”,
“device”:{”os”:”android”,”model”:”Pixel 7”,”geo”:{”lat”:39.98,”lon”:-82.98}},
“event”:{”name”:”demo”,”seq”:2,”ts”:”2026-02-17T14:40:26.189Z”}}

This data is being written to a json_bronze table, and is storing the user, email, and country fields as String and the nested fields as Struct types. It also adds Kafka metadata fields.

import dlt
from pyspark.sql.functions import current_timestamp, col, from_json, expr

SERVERS = "REDACTED"
TOPIC = “neil_struct_topic”

# Explicit JSON schema for predictable demos (no schemaLocationKey)
SCHEMA_DDL = “”“
  user STRING,
  email STRING,
  country STRING,
  device STRUCT<os: STRING, model: STRING, geo: STRUCT<lat: DOUBLE, lon: DOUBLE>>,
  event STRUCT<name: STRING, seq: BIGINT, ts: STRING>
“”“

@dlt.table(
    name=”json_bronze”,
    comment=”Raw Kafka payload with explicit JSON schema and rescued data”,
    table_properties={”quality”: “bronze”}
)
def json_bronze():
    df = (
        spark.readStream
            .format(”kafka”)
            .option(”kafka.bootstrap.servers”, SERVERS)
            .option(”kafka.security.protocol”, “SSL”)
            .option(”subscribe”, TOPIC)
            .option(”startingOffsets”, “earliest”)
            .load()
    )

    parsed = (
        df.selectExpr(”CAST(value AS STRING) AS json_str”, “topic”, “partition”, “offset”, “timestamp”)
          .select(
              from_json(
                  col(”json_str”),
                  SCHEMA_DDL,
                  options={”rescuedDataColumn”: “_rescued_data”}  # capture type mismatches/new fields
              ).alias(”data”),
              “topic”, “partition”, “offset”, “timestamp”
          )
          # data.* includes _rescued_data already; do not reselect it to avoid duplicate column error
          .selectExpr(”data.*”, “topic”, “partition”, “offset”, “timestamp AS kafka_timestamp”)
          .withColumn(”ingestion_ts”, current_timestamp())
    )

    return parsed

Below is the current state of our target table. This matches the state of the diagram above. Users 1 through 4 have been ingested into the target Delta table.

Now let’s imagine the nature of our topic changes and the device and event fields need to become flexible, allowing for new nested fields to be added anytime and reflected in our target. In the current implementation, these new fields would not appear automatically in our Struct columns.

One way to allow for flexibility of these nested columns is to update the Struct columns to Variant type. As mentioned above, however, Alter Table is unavailable on a streaming table to update column types. Here’s how to accomplish a streaming table update while ensuring there is no data-loss or duplication.

Step 1: Pause your pipeline

Whether your pipeline is continuous or scheduled to run periodically, you don’t want to be ingesting data while performing these actions.

Under the scheduled Job:

Step 2: Backup Bronze Table

This retains all data we’ve already ingested (users 1-4), including data that no longer exists in our Kafka topic (users 1-3). We’ve now ensured we won’t lose records that have aged out of the source.

CREATE TABLE neil_test_catalog.streaming.json_bronze_backup
SELECT * FROM neil_test_catalog.streaming.json_bronze;

Step 3: Determine Max Offset per Partition

At this point, Kafka has continued to receive new records behind the scenes (user-5 and user-6). We want to ensure that when we Full Refresh our pipeline, we read only these new messages without reprocessing data (user-4). This is achieved by making use of Spark’s readStream startingOffsets parameter.

This parameter allows you to specify which offsets Spark should begin reading data from, the first time a pipeline runs. Keep in mind that in Kafka, offsets are integers that uniquely identify messages per partition, so you’ll have to specify a starting offset for each partition in your topic. SDP uses these offsets to ensure that upon initial startup, the pipeline begins exactly where you intend. From that point forward, Spark continuously records these offsets in its internal checkpoints to track progress over time and guarantee exactly-once processing. It’s also a good idea to store this Kafka metadata in the Delta table itself.

Here’s the same snapshot of our source topic and target Delta table, with offset information included:

Notice that to begin processing at user-5 we will need to specify our pipeline start at offset 4. The startingOffsets parameter expects this information in the following format:

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)

If our topic contained multiple partitions, starting offsets must be set for each partition and would look like this for two partitions:

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4, “1”:6}}’)

But how can we find this information?

Finding via Metadata Columns (If Defined in Pipeline and Tracked in Delta Table)

If you’ve added Kafka metadata to your bronze table, you can retrieve your max offset per partition there. Remember that for this simple example we only have one Kafka partition. If using this method it’s important to note that these results show the most recent offset ingested, and +1 must be added to specify where Spark should start reading.

SELECT partition, MAX(offset)
FROM neil_test_catalog.streaming.json_bronze
GROUP BY partition

Finding via Spark Declarative Pipelines Checkpoints

Another way to retrieve your offset information is to query SDP’s /checkpoints/ folder that your pipeline uses to track state and progress. For more detailed information on checkpoints and how Spark Structured Streaming achieves exactly-once processing, check out this blog: Inside Delta Lake’s Idempotency Magic: The Secret to Exactly-Once Spark.

First, find your streaming table’s storage location via DESCRIBE DETAIL.

Note: If your table is Unity Catalog-managed, this method requires direct read access to the table’s managed storage location.

Using the location, you can append “/_dlt_metadata/checkpoints/your_table_name/” to find the most recent streaming query context (the greatest number). For this streaming table the max result is 8 as shown below.

path = “s3://....”
metadata_path = path + “/_dlt_metadata/checkpoints/neil_test_catalog.streaming.json_bronze/”
display(dbutils.fs.ls(metadata_path))

Within the numbered subfolder under your table name, you will see /offsets/ and /commits/ folders, each also containing numbered folders 0/, 1/, 2/ and so on. These folders represent streaming batches in SDP.

• offsets/N is a write‑ahead log entry written before processing batch N. It stores the end offsets (high‑water mark) for that batch — i.e., “read up to here” for each topic/partition.

• commits/N is only written after batch N has finished successfully.

Because an offsets/N entry can exist even if its corresponding commits/N is missing (a batch started but never committed), you should follow these steps to determine where to retrieve your startingOffsets.

List the batch IDs in both folders:

offsets_path = path + "/_dlt_metadata/checkpoints/your_table_name/8/offsets/"
commits_path = path + "/_dlt_metadata/checkpoints/your_table_name/8/commits/"
display(dbutils.fs.ls(offsets_path))
display(dbutils.fs.ls(commits_path))

Let:

• commit_batches = all numeric batch IDs under commits/

• max_commit_batch = largest value in commit_batches

Use max_commit_batch as the last fully committed batch, and read the matching offsets file:

Always read offsets/max_commit_batch to get the correct startingOffsets JSON.

Ignore any higher batch ID that appears only under offsets/ but not commits/. That batch started but never finished, so if you treat its offsets as your starting position, Spark will behave as if that data has already been read and will skip it rather than processing it.

In my example, both the /offsets/ and /commits/ folders contain only batches 0 and 1, so using max_commit_batch, we read startingOffsets from /offsets/1.

metadata_path = path + “/_dlt_metadata/checkpoints/neil_test_catalog.streaming.json_bronze/8/offsets/1/”
display(dbutils.fs.head(metadata_path))

{”batchWatermarkMs”:0,”batchTimestampMs”:1771439553201,”conf”:{...}}
{”neil_struct_topic”:{”0”:4}}

Notice the final line is our starting partition:offset information in the exact format we created manually above. Spark tracks the next offset to read, so you do not need to increment +1 via this method. Again, if our topic had multiple partitions it might look like:

{”neil_struct_topic”:{”0”:4, “1”:6}}

Step 4: Update pipeline definition

Now it’s time to apply the actual logic changes that prompted this process. For my example, I add variant support to table properties:

table_properties={”quality”: “bronze”, “delta.feature.variantType-preview”: “supported”}

And cast the Struct columns to Variant:

return (
        parsed
            .withColumn(”event”, expr(”parse_json(to_json(event))”))
            .withColumn(”device”, expr(”parse_json(to_json(device))”))
        )

Step 5: Full Refresh Table

With pipeline logic updated, it’s time to run with Full refresh to wipe the target table and ingest the new Kafka records starting at our specified offsets. The resulting table will contain only the records we did not back up via Step 2.

To do this, we run our pipeline with full refresh after adding our startingOffsets (line 28). Here is the final pipeline definition.

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)

import dlt
from pyspark.sql.functions import current_timestamp, col, from_json, expr

SERVERS = “REDACTED”
TOPIC = “neil_struct_topic”

# Explicit JSON schema for predictable demos (no schemaLocationKey)
SCHEMA_DDL = “”“
  user STRING,
  email STRING,
  country STRING,
  device STRUCT<os: STRING, model: STRING, geo: STRUCT<lat: DOUBLE, lon: DOUBLE>>,
  event STRUCT<name: STRING, seq: BIGINT, ts: STRING>
“”“

@dlt.table(
    name=”json_bronze”,
    comment=”Raw Kafka payload with explicit JSON schema and rescued data”,
    table_properties={”quality”: “bronze”, “delta.feature.variantType-preview”: “supported”}
)
def json_bronze():
    df = (
        spark.readStream
            .format(”kafka”)
            .option(”kafka.bootstrap.servers”, SERVERS)
            .option(”kafka.security.protocol”, “SSL”)
            .option(”subscribe”, TOPIC)
            .option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)
            .load()
    )

    parsed = (
        df.selectExpr(”CAST(value AS STRING) AS json_str”, “topic”, “partition”, “offset”, “timestamp”)
          .select(
              from_json(
                  col(”json_str”),
                  SCHEMA_DDL,
                  options={”rescuedDataColumn”: “_rescued_data”}  # capture type mismatches/new fields
              ).alias(”data”),
              “topic”, “partition”, “offset”, “timestamp”
          )
          # data.* includes _rescued_data already; do not reselect it to avoid duplicate column error
          .selectExpr(”data.*”, “topic”, “partition”, “offset”, “timestamp AS kafka_timestamp”)
          .withColumn(”ingestion_ts”, current_timestamp())
    )

    return (
        parsed
            .withColumn(”event”, expr(”parse_json(to_json(event))”))
            .withColumn(”device”, expr(”parse_json(to_json(device))”))
        )

On the pipeline page:

Here’s the result: user-5 and user-6 as expected:

Step 6: Insert Historical Records

To complete our intended result table, insert historical data from the backup table, ensuring the data matches the new table format (cast columns, etc.):

INSERT INTO neil_test_catalog.streaming.json_bronze
SELECT user, email, country, to_variant_object(device), to_variant_object(event), _rescued_data, topic, partition, offset, kafka_timestamp, ingestion_ts
FROM neil_test_catalog.streaming.json_bronze_backup

Our final result. Your pipeline may be resumed.

user:string
email:string
country:string
device:variant
event:variant
_rescued_data:string
topic:string
partition:integer
offset:long
kafka_timestamp:timestamp
ingestion_ts:timestamp

Step 7: Revert startingOffsets

To avoid pipeline failures in the case of a future Full Refresh, revert startingOffsets to its prior value.

.option(”startingOffsets”, “earliest”)

Frequently Asked Questions

Q: Why can’t I just run a standard Full Refresh on the pipeline?

A: A Full Refresh clears the target table and re-reads the source from the beginning. If your Kafka topic has a retention policy (TTL), data that has "aged out" of the topic will be permanently lost because it no longer exists in the source to be re-read.

Q: Why is startingOffsets necessary if I have a backup?

A: While the backup saves your history, startingOffsets ensures your pipeline resumes reading exactly where the backup stopped. Without this explicit instruction, the pipeline might default to "earliest" (reading only what remains in Kafka, creating a gap) or "latest" (skipping data that arrived during the maintenance window).

Q: Is this process required for "Append-Only" tables?

A: Generally, yes, if you need to restructure the existing table. If you are only adding new columns that are nullable, you might rely on schema evolution, but fundamental type changes usually require the table to be rewritten.

As a Specialist Solutions Architect at Databricks, I often hear the same questions from customers who are migrating dashboards from legacy BI tools to Databricks AI/BI Dashboards:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

In the first blog post in this series, I focused on the first two questions and showed how to recreate:

• context filters using parameters in dataset SQL, and

• “Only Relevant Values” filters using field filters and query-based parameters.

This blog tackles the third question: “How do we replace filter actions from existing dashboards when we click on a bar/segment/point?”

In many BI tools, this behavior is configured as a filter action (a click on a mark filters other views). In Databricks AI/BI Dashboards, the equivalent interactivity is split into two built-in features:

• Cross-filtering, where clicking a mark in one chart filters other charts on the same page that use the same dataset.

• Drill-through, where right-clicking a mark opens a target page filtered to that selection.

Once you combine cross-filtering and drill-through with the context and cascading patterns, you can reproduce most real-world filter-action workflows.

As before, I’ll use the built-in samples.tpch dataset. I’ve also published the companion dashboard, so you can follow along and inspect the configurations yourself.

Recap: TPCH Sales dataset

To keep examples concrete, we’ll use the TPCH sample data that ships with Databricks in the samples.tpch schema. I’ll reuse the same base dataset, TPCH Sales, from the first blog post, which joins tables region, nation, customer, orders, and lineitem, and computes revenue:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

In the companion dashboard, this is the TPCH Sales dataset. It and its derivatives are used by multiple pages:

• Context filter

• Cascading filters with field filters

• Cascading filters with query-based parameters

• Cross-filtering (new in this post)

• Drill-through details (new in this post)

• and others

Filter actions vs. cross-filtering and drill-through

Before we build anything, it helps to align vocabulary.

In many traditional BI tools, filter actions are configured explicitly:

• You specify one or more source sheets.

• You specify one or more target sheets.

• You choose which fields are passed as filters.

• You choose what happens when the selection is cleared.

In Databricks AI/BI Dashboards, cross-filtering is implicit. You don’t turn it on in the visualization panel.

• It is automatically applied to supported visualization types that use the same dataset.

• When you click a bar, slice, or point, AI/BI adds a filter based on that value and re-runs all other visualizations on the page that share the dataset.

In Databricks AI/BI Dashboards, drill-through is also implicit, but slightly more structured:

• When you right-click a supported chart type, AI/BI shows Drill to → <target page> if there is another page in the dashboard that uses the same dataset.

• The target page opens with all visuals based on that dataset filtered to the selected segment, and any compatible filters on that dataset are auto-populated.

Conceptually:

• Cross-filtering ≈ a within-page filter action.

• Drill-through ≈ a navigation filter action (summary → details).

The rest of the post walks through how to configure your pages so these implicit behaviors “just work”.

1. Recreating within-page filter actions with cross-filtering

A very common dashboard pattern is:

• A summary bar chart (for example, total revenue by nation).

• One or more supporting charts (for example, revenue share by region, or a donut chart for mix).

• A filter action so clicking a bar filters the other visuals.

In AI/BI Dashboards, this becomes cross-filtering on top of the TPCH Sales dataset.

1.1. When cross-filtering is applied

Cross-filtering is applied automatically when all of the following are true:

• The visualizations are on the same page.

• The visualizations use the same dataset (for example, TPCH Sales).

• The visualization type is one of the supported chart types: Bar, Box plot, Heatmap, Histogram, Pie, Scatter, or Point map.

If those conditions are met, there is nothing to enable. Clicks on supported charts become filters for any other visualizations on the page that use that dataset.

1.2. Build the Cross-filtering page

In your AI/BI dashboard, add a page named Cross-filtering.

On this page:

1. Add a bar chart: Revenue by nation

   • Visualization: Bar

   • Dataset = TPCH Sales

   • X axis: nation

   • Y axis: SUM(revenue)

2. Add a pie: Revenue by region

   • Visualization: Pie

   • Dataset = TPCH Sales

   • Slice by (Color): region

   • Value (Angle): SUM(revenue)

3. Add filters (optional)

   • Add a Region filter on TPCH Sales.region

   • Add a Nation filter on TPCH Sales.nation

These are just standard filter widgets. There is no “cross-filtering” toggle anywhere in the configuration.

In the companion dashboard, this page is already built for you.

1.3. Use cross-filtering on the charts

Try the following workflow:

1. Select two regions, AFRICA and ASIA, on the Revenue by region chart.

2. Click the bar for JAPAN in the Revenue by nation chart.

Because cross-filtering is automatic for supported charts that share a dataset:

• AI/BI adds filters Nation: JAPAN and Region: AFRICA, ASIA to the TPCH Sales dataset for this page.

• Both charts get updated accordingly.

You can Reset all to default and try a different filter combination.

From a migration perspective, this answers a common question: “Can clicking a bar automatically update the rest of the dashboard in AI/BI?” Yes – when charts share a dataset and use supported visualization types, cross-filtering works implicitly with no additional configuration.

2. Recreating across-page filter actions with drill-through

Another classic summary → detail filter-action pattern is:

• A summary view (for example, revenue by nation).

• A detail view (for example, individual orders).

• A filter action that passes the selected value into the detail sheet as a filter.

In AI/BI Dashboards, this is implemented as drill-through.

2.1. How drill-through is applied

Drill-through shows up as a right-click option when several conditions are satisfied.

• The source chart is a supported type: Bar, Box plot, Heatmap, Histogram, Pie, Scatter, or Point map.

• There is at least one target page in the same dashboard where:

  • At least one visualization uses the same dataset as the source chart.

  • The field you click on has a compatible filter or column on the target page.

In recent AI/BI releases, drill-through no longer requires an explicit target filter; any visualization based on the same dataset as the source selection is filtered automatically, and filters (if they exist) are populated with the drilled values.

There is no drill-through toggle in the widget side panel. Once the above conditions are true, AI/BI surfaces Drill to → <page name> in the context menu automatically.

2.2. Build the Drill-through details page

In the same dashboard, add another page named Drill-through details.

On this page:

1. Add a detail table visualization

   • Visualization: Table

   • Dataset = TPCH Sales

   • Columns: region, nation, customer_id, customer_name, order_id, order_date, revenue

2. Add filters (optional)

   • Add a Region filter on TPCH Sales.region

   • Add a Nation filter on TPCH Sales.nation

   • Add a Customer filter on TPCH Sales.customer_id

Again, there is no special drill-through configuration here – just a normal page that uses the same dataset.

In the companion dashboard, this page is already built for you.

2.3. Drill from summary to details

Go back to the Cross-filtering page and right-click a bar in the Revenue by nation chart (for example, UNITED STATES).

After clicking Drill to → Drill-through details, AI/BI opens the Drill-through details page and filters all visualizations based on TPCH Sales to Nation: UNITED STATES.

From the user’s perspective, this feels almost identical to a filter action in traditional dashboards that navigates from a summary sheet to a detailed sheet with the selected country carried over.

Cross-filtering vs. drill-through vs. filter widgets

Across this blog and the first blog, you now have three main interaction tools in AI/BI Dashboards:

1. Filter widgets

   • Context filters, cascading filters, and query-based parameters.

   • Best for primary, always-visible controls like Region, Date, Product.

2. Cross-filtering

   • Click data points in a supported visualization to filter other charts on the page that use the same dataset.

   • Best for ad-hoc exploration, answering questions like “What happens if I focus only on this region?” and “Which nations are driving that spike?”.

3. Drill-through

   • Right-click a mark to open another page already filtered to that selection.

   • Best for guided summary-to-detail flows where you don’t want to cram everything onto one page.

A simple migration rule of thumb:

• Use filter widgets to rebuild the core filter panels from your existing dashboards.

• Use cross-filtering to automatically filter other visualizations on the page based on chart interactions.

• Use drill-through to replace “go to sheet” filter actions and connect high-level KPIs to detail pages.

Summary and what’s next

In this second blog post in the series, we answered: “Do you support filter actions when I click on a bar or a point?”

The short answer is yes:

• Cross-filtering lets viewers click on supported charts to filter all other visualizations on the same page that share a dataset – no configuration required.

• Drill-through lets viewers right-click a mark and open another page where visuals on the same dataset are already filtered to the selected values, and any matching filters are pre-populated.

Combined with the patterns from the first blog post – context filters and cascading “Only Relevant Values” filters – you now have a robust toolkit for recreating the interactive filtering experience your users expect from traditional dashboards inside Databricks AI/BI Dashboards.

The companion dashboard now includes: a context filter page, a field-based cascading page, a query-based cascading page, a cross-filtering page, and a drill-through details page. You can import it into your workspace and adapt the patterns to your own datasets.

In the next post in this series, I’ll tackle the remaining migration question: “How do we do user-based filtering and row-level security in AI/BI Dashboards?”

As a Specialist Solutions Architect at Databricks, I regularly work with customers who are migrating critical analytics from existing BI tools to Databricks AI/BI Dashboards – and the first questions I usually get are about filters.

Teams want to know:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

These aren’t cosmetic features. They’re how analysts actually interact with dashboards, and they’re often the reason an existing BI dashboard feels “alive” instead of static.

In this post, I’ll walk through how to implement two familiar filter patterns from existing BI dashboards in Databricks AI/BI Dashboards, using the built-in samples.tpch dataset:

1. Context filters → implemented as parameters in dataset SQL

2. “Only Relevant Values” or cascading filters → implemented with field filters and query-based parameters

Row-level security and user-based filtering deserve their own deep dive, and action-style interactions (cross-filtering and drill-through) could easily fill another post, so I’ll cover those separately.

I’ve also published the companion dashboard, so you can follow along and inspect the configurations yourself.

Quick primer: datasets and filters in Databricks AI/BI Dashboards

Before we map those patterns, it helps to align on a few AI/BI Dashboards concepts:

Datasets

In AI/BI Dashboards, each dashboard has a Data tab where you define one or more datasets:

• A dataset is defined by an SQL query, direct reference to a Unity Catalog table/view, or an uploaded file.

• Multiple visualizations can reuse the same dataset.

• Datasets are bundled with the dashboard when you share/export/import it.

Practically, a dataset is your “model” for a set of visuals: one query, many charts.

Field filters vs parameter filters

AI/BI Dashboards support two core ways to filter data from a dashboard: field filters and parameter filters. Both are implemented as filter widgets, but they behave differently under the hood.

Field filters are applied directly to dataset fields (columns) on top of the dataset query. Processing behaviour is defined by the dataset performance thresholds. Specifically, for small datasets (≤ 100K rows or ≤ 100MB), results are pulled to the browser and visualization-specific filtering and aggregation are applied client-side. For larger datasets, Databricks wraps the dataset query in a WITH clause and applies the filter predicates and aggregations in Databricks SQL warehouse (DBSQL).

Parameter filters are applied to parameters, which are variables that get substituted into your dataset SQL at runtime. When the parameter value changes, the query is always re-run in DBSQL.

In other words, field filters operate on the results of the dataset query, while parameter filters operate inside the dataset SQL itself.

To speed up processing, various caching layers in AI/BI Dashboards and DBSQL are used.

We’ll use parameter filters to emulate context filters, and field filters + query-based parameters to emulate “Only Relevant Values.”

Filter scope: global, page-level, and widget-level

Filters in AI/BI Dashboards also differ by scope:

• Global filters are interactive filters in the global filters panel that apply across all pages of the dashboard to any visualization that shares the selected datasets.

• Page-level filters are interactive filter widgets placed on a specific page in the canvas. They apply to all visualizations on that page that share one or more datasets.

• Widget-level filters are static filters configured directly on a single visualization widget in its configuration panel. Authors set the values, and viewers can’t change them.

With that foundation in place, we can now map these context filters and “Only Relevant Values” patterns into AI/BI Dashboards patterns.

Sample dataset: TPCH on Databricks

To keep examples concrete, we’ll use the TPCH sample data that ships with Databricks in the samples.tpch schema.

For the purposes of this post, you can start by creating a dataset that joins tables region, nation, customer, orders, and lineitem:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

In AI/BI Dashboards, you define this query as a dataset in the Data tab and then reuse it across multiple visualizations. Let’s call this dataset TPCH Sales.

We’ll reuse this same dataset or its derivatives throughout the rest of the post to illustrate context filters and cascading filters.

Implementing context filters with parameters in dataset SQL

What a context filter does

A context filter defines a high-level subset of the data:

• The context filter is applied first, often materializing a temporary subset.

• Other filters and some calculations are then evaluated on top of that subset.

Context filters are used to:

• Improve performance by filtering early and shrinking the working set.

• Enforce logical order, such as “always filter by Region first.”

• Make other filters depend on that subset.

How to think about context in AI/BI Dashboards

Given the primer:

• Field filters operate on the results of the dataset query (Databricks wraps your dataset SQL and applies them on top).

• Parameter filters substitute values directly into your dataset SQL, so they filter inside the query, before joins and aggregations.

If you want “context” behavior – filter first, then apply everything else – you should implement that filter as a parameter in the dataset SQL, driven by a parameter filter widget.

Pattern: treat the context as a base parameter

Let’s add a context filter for Region:

If you’re following along with the companion dashboard, this setup lives on the “Context filter” page.

Step 1. Define TPCH Sales (Context) with a Region parameter

Create a dataset TPCH Sales (Context):

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey
WHERE r.r_name = :region_param      -- “context” filter

In the dataset’s Parameters panel:

• Define region_param with type String.

• Optionally set a default (for example, AMERICA) so the dataset runs without any dashboard filter.

This makes region_param the context for all visuals that use TPCH Sales (Context).

Step 2. (Optional but nice) Create a helper dataset for Region values

You can drive region_param directly from TPCH Sales (Context), but a tiny helper dataset keeps things tidy, convenient, and re-usable.

Create TPCH Regions (Context):

SELECT DISTINCT r_name AS region
FROM samples.tpch.region
ORDER BY region;

This dataset has no parameters; it just returns the list of available regions.

Step 3. Add a Region parameter filter widget

We will configure the widget as a page-level filter (alternatively, you can move it into the global filters panel if it should apply across pages).

On the page where you want Region as a context filter:

1. Add a filter widget and title it Region.

2. Set the filter type to Single value.

3. Configure it as a parameter filter:

   • Fields: TPCH Regions (Context).region

   • Parameters: TPCH Sales (Context).region_param

If you don’t want a helper dataset, you can instead use TPCH Sales (Context).region as the field source, but the wiring is otherwise identical.

Effect

• When a viewer selects Region: EUROPE in the Region filter:

  • The widget writes “EUROPE“ into region_param for TPCH Sales (Context).

  • TPCH Sales (Context) reruns with WHERE r.r_name = ‘EUROPE’.

• All visuals built on TPCH Sales (Context) now use only European data as their starting point.

• Any additional field filters (for example, Nation, Customer, Date) operate on this already-filtered subset, just like secondary filters evaluated after a context filter.

• From the viewer’s perspective, Region behaves like a true context filter: it defines the base subset of data first, and everything else – other filters, cross-filtering, drill-through – is evaluated on top of that context.

Implementing “Only Relevant Values” with cascading filters and query-based parameters

What “Only Relevant Values” behavior means

“Only Relevant Values” behavior on a filter shrinks the list of values based on the current state of other filters and the view:

• If you select Region: ASIA, the Country filter only shows countries that actually have data in ASIA.

• As you add more filters, each filter’s domain is recomputed from the filtered dataset.

Practically, this gives you cascading filters that stay in sync with each other and with the current slice of data.

How to think about “Only Relevant Values” in AI/BI Dashboards

In AI/BI Dashboards, you get the same effect in two ways:

1. Field filters on the same dataset – AI/BI recomputes the value list based on the current filtered dataset.

2. Query-based parameters – a specialized filter widget that both populates its values from a query, and writes the selected value into a parameter used in your dataset SQL.

Pattern 1: Cascading filters with field filters

The simplest way to mimic “Only Relevant Values” is to use field filters wired to the same dataset. AI/BI Dashboards will automatically recompute each filter’s value list based on the current filtered dataset.

We’ll build a Region → Nation → Customer cascade on top of TPCH Sales (Cascading Pattern 1).

In the companion dashboard, this pattern is implemented on the “Cascading filters with field filters” page.

Step 1. Define TPCH Sales (Cascading Pattern 1)

Create a dataset TPCH Sales (Cascading Pattern 1) with the base TPCH join and a revenue metric:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

This dataset has no parameters; all filtering will be done with field filters on top of the query results.

Step 2. Add Region, Nation, and Customer field filters

On the dashboard page where you want cascading behavior:

1. Add three field filter widgets with titles Region, Nation, and Customer.

2. Configure each widget as a page-level filter (or move them into the global filters panel if you want them to apply across pages).

3. Connect the filters to the following fields from TPCH Sales (Cascading Pattern 1):

   • Region → region

   • Nation → nation

   • Customer → customer_id

No parameters are involved here – these are pure field filters on a single dataset.

Effect

• When a viewer selects region: ASIA, the TPCH Sales (Cascading Pattern 1) dataset is filtered to ASIA for all visuals on the page.

• The Nation field filter’s value list is recomputed from that filtered dataset, so it only shows nations in ASIA.

• After the viewer chooses a nation (e.g., JAPAN), the Customer field filter shrinks to show only customers in that nation.

• From the user’s perspective, these field filters behave like filters with an “Only Relevant Values” option enabled: each dropdown shows only values that exist in the currently filtered data. Under the hood, AI/BI Dashboards are simply applying field filters on top of a single dataset and recomputing the dropdown values from the currently filtered result set.

Pattern 2: Cascading filters with query-based parameters

In Pattern 1, we used field filters only. In some cases you may want more control over how dropdown values are loaded, or you may want the same parameter to drive multiple datasets. In that case you can use query-based parameters. A query-based parameter filter widget gets its dropdown values from a field in a “choices” dataset, and writes the selected value into one or more parameters that are used in dataset SQL.

Here we’ll build a three-level cascade Region → Nation → Customer using:

• One main dataset: TPCH Sales (Cascading Pattern 2)

• Three small “value list” datasets:

  • TPCH Regions (Cascading Pattern 2)

  • TPCH Nations by Region (Cascading Pattern 2)

  • TPCH Customers by Nation (Cascading Pattern 2)

In the companion dashboard, this pattern is implemented on the “Cascading filters with query-based parameters” page.

Step 1. Define TPCH Sales (Cascading Pattern 2)

Create the TPCH Sales (Cascading Pattern 2) dataset with parameters for region, nation, and customer:

SELECT
 r.r_name              AS region,
 n.n_name              AS nation,
 c.c_custkey           AS customer_id,
 c.c_name              AS customer_name,
 o.o_orderkey          AS order_id,
 o.o_orderdate         AS order_date,
 l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey
WHERE (:region_param   = 'All' OR r.r_name = :region_param)
  AND (:nation_param   = 'All' OR n.n_name = :nation_param)
  AND (:customer_param = 0     OR c.c_custkey  = :customer_param);

In the dataset’s Parameters panel:

• Set region_param type to String.

• Set nation_param type to String.

• Set customer_param type to Numeric / Integer (to match c_custkey).

This last bit is important: the Customer filter uses a numeric field, so the parameter must be numeric as well.

Step 2. Create helper datasets for the dropdowns

1. TPCH Regions (Cascading Pattern 2) – list of regions:

SELECT DISTINCT r_name AS region
FROM samples.tpch.region
ORDER BY region;

2. TPCH Nations by Region (Cascading Pattern 2) – nations for the selected region:

SELECT DISTINCT n.n_name AS nation
FROM samples.tpch.nation   AS n
JOIN samples.tpch.region   AS r ON n.n_regionkey = r.r_regionkey
WHERE r.r_name = :region_param
ORDER BY nation;

This dataset defines its own region_param (string) in the Data tab.

3. TPCH Customers by Nation (Cascading Pattern 2) – customers for the selected nation:

SELECT DISTINCT
  c.c_custkey AS customer_id
FROM samples.tpch.nation   AS n
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
WHERE n.n_name = :nation_param
ORDER BY customer_id;

This dataset defines nation_param (string). customer_id is numeric, matching customer_param in TPCH Sales (Cascading Pattern 2).

Run each dataset in the Data tab once to confirm they succeed.

Step 3. Add Region, Nation, and Customer filter widgets

On your dashboard page, add three filter widgets and wire them to fields and parameters. Configure all three widgets as page-level filters (or move them into the global filters panel if they should apply across pages).

1. Region filter widget

• Filter type: Single value

• Fields: TPCH Regions (Cascading Pattern 2).region

• Parameters:

  • TPCH Sales (Cascading Pattern 2).region_param

  • TPCH Nations by Region (Cascading Pattern 2).region_param

• Default value: All

This keeps region_param in TPCH Sales (Cascading Pattern 2) and TPCH Nations by Region (Cascading Pattern 2) in sync.

2. Nation filter widget

• Filter type: Single value

• Fields: TPCH Nations by Region (Cascading Pattern 2).nation

• Parameters:

  • TPCH Sales (Cascading Pattern 2).nation_param

  • TPCH Customers by Nation (Cascading Pattern 2).nation_param

• Default value: All

This keeps nation_param in TPCH Sales (Cascading Pattern 2) and TPCH Customers by Nation (Cascading Pattern 2) in sync.

3. Customer filter widget

• Filter type: Single value

• Fields: TPCH Customers by Nation (Cascading Pattern 2).customer_id

• Parameters: TPCH Sales (Cascading Pattern 2).customer_param

• Default value: 0

Effect

• When a viewer selects Region: AMERICA:

  • The Region widget writes “AMERICA“ into region_param in TPCH Sales (Cascading Pattern 2) and TPCH Nations by Region (Cascading Pattern 2).

  • TPCH Nations by Region (Cascading Pattern 2) reruns and returns only nations in AMERICA, so the Nation dropdown only shows those nations.

• When the viewer then selects Nation: UNITED STATES:

  • The Nation widget writes “UNITED STATES“ into nation_param in TPCH Sales (Cascading Pattern 2) and TPCH Customers by Nation (Cascading Pattern 2).

  • TPCH Customers by Nation (Cascading Pattern 2) reruns and returns only customers in UNITED STATES, so the Customer dropdown only shows those customer IDs.

• When the viewer selects a specific Customer (for example, 607):

  • The Customer widget writes 607 into customer_param in TPCH Sales (Cascading Pattern 2).

  • TPCH Sales (Cascading Pattern 2) reruns with region_param, nation_param, and customer_param applied, and all visuals built on this dataset show only orders for customer 607 in UNITED STATES / AMERICA.

• From the viewer’s perspective, Region → Nation → Customer behaves like cascading filters with “Only Relevant Values” behavior enabled. Under the hood, each dropdown is a query-based parameter filter, and the Region and Nation widgets keep parameters in multiple datasets in sync, while Customer filters the main TPCH Sales (Cascading Pattern 2) dataset down to a single customer.

Which pattern when?

Both patterns get you “Only Relevant Values”-style cascading behavior, but they shine in different situations.

Pattern 1 – Cascading field filters

Use this when:

• You’re working off one main dataset per page.

• You want the simplest authoring experience: add field filters, connect them to the dataset, done.

• “Allow All” and easy clearing of filters are important to your users.

This is the closest to what many BI tools do by default and is usually the right starting point.

Pattern 2 – Cascading query-based parameters

Use this when:

• You need parameters that drive multiple datasets.

• You want tighter control over dropdown values, including custom queries per level.

• You’re comfortable managing parameter types and wiring filters to multiple datasets.

Pattern 2 is more flexible and explicit, but also more advanced. In practice, I start with Pattern 1 for most dashboards, and reach for Pattern 2 when I need parameter-driven logic or want to reuse the same parameters across several datasets and pages.

Summary

In this post, we looked at how to carry two of the most important filter patterns from traditional BI dashboards into Databricks AI/BI Dashboards:

• Context filters become parameters in your dataset SQL, driven by parameter filter widgets. This lets you enforce “filter by Region first” semantics and shrink the working set before joins and aggregations.

• “Only Relevant Values” / cascading filters can be implemented either with simple field filters on a single dataset (Pattern 1) or with query-based parameters and helper datasets (Pattern 2) when you need more control and reusable parameters.

The companion dashboard includes all three examples: a context filter page, a field-based cascading page, and a query-based cascading page. You can import it into your workspace and adapt the patterns to your own datasets.

In future posts, I plan to cover:

• Row-level security and user-based filtering in AI/BI Dashboards

• Action-style interactions such as cross-filtering and drill-through in AI/BI Dashboards

If you’re starting a migration from an existing BI tool to Databricks AI/BI today, I recommend:

1. Identify your key context filters (Region, Business Unit, etc.) and implement them as parameters in dataset SQL.

2. Start with Pattern 1 (field filters) for cascading behavior, and only move to Pattern 2 where you truly need parameter-driven logic or shared parameters across datasets.

These two patterns alone are usually enough to make an AI/BI dashboard feel as interactive and “alive” as the dashboards your teams are used to today.

The goal was simple but demanding: near-real-time data availability and consistently fast query performance across large time ranges, even in the presence of late-arriving data and massive daily ingestion volumes.

Why Traditional Partitioning Fell Short and How Liquid Clustering Solves It

The original architecture relied on continuous streaming ingestion into Bronze tables from Kafka, followed by scheduled batch jobs that populated optimized Silver tables.Bronze tables were partitioned by date, while Silver tables were partitioned and z-ordered by relevant keys. When everything arrived on time and tables were fully optimized, query performance was excellent.

The problem began with late-arriving data.

Some events arrived days—even weeks—after their original event time. These records landed in partitions that had already been optimized, slowly reintroducing small, unoptimized files into previously clean partitions. Over time:

• File counts ballooned

• Queries that once ran in ~10 seconds stretched to a minute or more

• Performance degraded steadily as more late data accumulated

The only way to recover performance was to re-optimize entire partitions repeatedly, which became increasingly expensive and time-consuming at scale. Rigid partition boundaries simply did not work well with unpredictable arrival patterns.

This is where Liquid Clustering became a natural fit. Instead of relying on static partitions, Liquid Clustering incrementally maintains data layout quality as data arrives. It continuously rebalances files based on clustering keys, reducing the need for repeated full rewrites and making late-arriving data far less disruptive.

Liquid Clustering addresses these issues with a multi-dimensional, incremental clustering strategy. It removes rigid partition boundaries, continuously reorganizes poorly clustered segments, and supports both eager clustering (during ingestion) and lazy clustering (post-ingestion). Using a tree-based multi-column clustering, it improves data skipping and maintains predictable, low-latency query performance even on large, late-arriving datasets.

Where Liquid Met Production Reality: Scaling Challenges and Fixes

While Liquid Clustering addressed the core architectural issue, the migration itself surfaced a new set of challenges primarily driven by scale.

High-Throughput Ingestion and Backfills

One of the first challenges during the migration to Liquid Clustering was handling the existing historical data. To adopt the new layout strategy, tens of terabytes of data per day over several months had to be backfilled and reorganized. This was challenging because the platform also had to continue processing new streaming data and maintain Silver tables for analysts. Running backfill and OPTIMIZE jobs simultaneously put the system under extreme load, pushing cluster resources to their limits.

Long-Running OPTIMIZE Jobs After Enabling Liquid

After enabling Liquid Clustering, OPTIMIZE runtimes increased noticeably—not because of Liquid alone, but due to the scale at which it was introduced. Large historical backfills were running alongside ongoing ingestion, forcing OPTIMIZE to process very large data volumes under heavy skew. Certain clustering stages ended up handling disproportionate amounts of data, resulting in large shuffles, disk spills, and increasingly unpredictable runtimes.

To address this, eager clustering was enabled for streaming writes, moving a portion of the clustering work into ingestion. This reduced the amount of reorganization required during OPTIMIZE and helped stabilize optimization runtimes, especially once batch sizes were increased and clustering work was better distributed across the cluster.

In addition, Liquid-specific tuning played a critical role in improving OPTIMIZE stability at scale. Key adjustments included:

• Enhanced data skipping was enabled to reduce unnecessary data movement during clustering, significantly lowering shuffle volume and improving OPTIMIZE efficiency.

• Increased clustering parallelism was also configured to distribute clustering work more evenly across the cluster, reducing skew and stabilizing runtimes for large and wide tables.

For workload-specific tuning and the exact configuration details, we recommend reaching out to Databricks, as the optimal settings can vary based on data volume and cluster characteristics.

To further reduce reliance on manual OPTIMIZE jobs, we also leveraged Predictive Optimization (PO) to automatically manage optimization workloads—this topic is discussed in more detail in the upcoming section.

Eager Clustering at Scale: Small File Challenges

While eager clustering reduced the amount of work during OPTIMIZE, it introduced a new challenge when streaming batches were too small. This was especially noticeable during large historical backfills, where each batch was around 40 GB. Although each batch was locally clustered, the small size meant that OPTIMIZE still had to rewrite many small files, resulting in high write amplification, longer runtimes, and increased operational overhead.

Solution: Batch Size as a Critical Lever

One of the most important lessons from this migration was how sensitive eager clustering is to batch size, particularly for backfills. Increasing batch sizes to larger, more meaningful units—around 1 TB per batch—changed the behavior dramatically (particularly during petabyte-scale backfills) . Larger batches allowed eager clustering to produce larger, better-clustered files upfront, which significantly reduced or even eliminated downstream OPTIMIZE work. This not only shortened OPTIMIZE runtimes but also lowered overall operational overhead by triggering fewer jobs and improving system stability during high-volume backfill processing.

Infrastructure Constraints

Another challenge emerged from the existing cluster configuration. At petabyte-scale, OPTIMIZE planning occasionally failed due to driver disk exhaustion, caused by large Spark event logs generated during complex optimization planning. The original cluster setup, designed for partitioned tables, was no longer sufficient to handle the heavy resource demands of Liquid Clustering and large backfills.

Solution: Reducing Event Log Volume

The issue was mitigated by tuning Spark to limit event log growth during OPTIMIZE planning. This significantly reduced the size of driver-side logs, preventing disk exhaustion and allowing large optimization jobs to complete reliably even while ingestion workloads continued to run.

For workload-specific tuning and exact configuration details, we recommend reaching out to Databricks.

Cluster and Runtime Constraints

The existing cluster configuration was no longer sufficient to handle simultaneous high-volume ingestion and large-scale OPTIMIZE jobs. Under petabyte-scale workloads, resource contention could slow down processing and introduce instability.

Solution: Right-Sizing Compute and Updating Runtime

To address this, cluster capacity was increased where needed, reducing reliance on spot instances and favoring on-demand workers for stability during long-running OPTIMIZE operations. In addition, the latest DBR 17.3 runtime was adopted for all new Liquid tables, leveraging improvements that enhanced performance, stability, and optimization efficiency at scale.

Cluster capacity was adjusted to reduce reliance on spot instances for long-running OPTIMIZE jobs, favoring on-demand workers where stability mattered most.

Predictive Optimization: Powerful, but Needs Guardrails

Once we stabilized OPTIMIZE runtimes and tuned eager clustering, the next focus was on reducing reliance on manual optimization jobs. For this, we leveraged Predictive Optimization , which automatically determines when and how to run clustering operations based on table state and data layout.

Concurrent manual OPTIMIZE jobs sometimes conflicted with PO runs in a few cases , causing transaction failures. Limited observability made it difficult to track PO activity and determine how much data remained unoptimized, which could lead to degraded query performance.

To address this, we implemented several best practices:

• Monitor PO activity using system.storage.predictive_optimization_history to track execution and outcomes.

• Fallback manual OPTIMIZE jobs are run whenever PO does not execute successfully.

Results: Performance, Freshness, and Simpler Operations

The migration to Liquid Clustering delivered clear, measurable improvements:

• Query performance improved dramatically—around 50% faster for most queries and up to 90% faster for long-range scans

• File counts were cut roughly in half, reducing I/O overhead

• Data freshness improved from hours to minutes, enabling near-real-time analytics

• Operational overhead dropped significantly with UC-managed tables and automated optimization

• Legacy partitioning was eliminated, reducing technical debt and modernizing the architecture
  

Final Thoughts

Liquid Clustering fundamentally changes how data layout is managed at scale. By moving away from rigid partitions and embracing incremental, adaptive clustering, it becomes possible to handle late-arriving data, massive ingestion volumes, and evolving schemas without sacrificing performance or driving up costs.

The key is understanding the operational nuances: batch sizing, clustering strategy, Liquid-specific tuning, cluster configuration, and optimization automation. When those pieces come together, Liquid Clustering can unlock faster queries, fresher data, and a far more resilient data platform.

For readers interested in diving deeper:

• Arctic Wolf’s Liquid Clustering Architecture Tuned for Petabyte Scale – Databricks Blog

• Optimizing Delta Lake Tables with Liquid Clustering – Canadian Data Guy

As a Specialist Solutions Architect at Databricks, over the past year I’ve led the Databricks SQL Cost & Performance Optimization Assessment initiative, reviewing and tuning two dozen customer warehouses across industries such as ad tech, healthcare, fintech, and energy. Each engagement shared the same goal — improving performance, reducing cost, and uncovering hidden inefficiencies. While every environment is unique, several recurring themes emerged. The following ten lessons highlight recurring optimization patterns and practical takeaways that helped customers achieve measurable cost and performance improvements. Of course, these are generalizations — exceptions may apply depending on workload characteristics, data layout, and other factors.

1. Liquid Clustering effectively saves compute and improves performance

This should be foundational, yet I continue to see multi-terabyte tables without any data-layout optimization, or with Liquid Clustering misused as hierarchical sorting. In some cases, tables are clustered on too many columns, degrading clustering efficiency. In both scenarios, file pruning is limited, and queries end up scanning far more data than necessary.

Lesson: Use Liquid Clustering strategically to improve query filtering and data skipping. Follow Databricks best practices when defining clustering keys.

2. Predictive Optimization delivers ongoing performance and cost gains

Predictive Optimization automatically improves managed Delta tables by compacting files, applying Liquid Clustering, collecting statistics, and deleting old files, reducing both compute and storage costs. It continuously observes workloads and schedules these maintenance operations automatically.

Lesson: Enable Predictive Optimization on Unity Catalog managed tables to benefit from continuous layout tuning and automatic vacuuming — improving performance and lowering storage spend with no manual jobs to manage.

3. Missing statistics are a common performance issue

Across most environments, I found tables with missing or partial column statistics, often due to external table ingestion or infrequent maintenance. Without accurate statistics, the optimizer can’t effectively estimate data volumes or apply efficient file pruning.

Lesson: Run ANALYZE TABLE regularly or rely on Predictive Optimization to automate stats collection for managed Delta tables. Missing statistics can significantly degrade query performance.

4. Disk spills are silent performance killers

Between 5% and 20% of queries in typical environments spill to disk during shuffles, adding seconds or even minutes to query runtime. These issues often go unnoticed because queries still “succeed,” but at a high cost.

Lesson: Identify queries with significant disk spill by analyzing the query history system table. Rewriting a query, adding repartitioning or join hints, splitting large queries into smaller ones that process subsets of data, or increasing warehouse size are all valid strategies to mitigate spills. Reducing disk spills alone can yield a 10–30% performance improvement.

5. Tiny files are the hidden tax of inefficient ingestion

One customer’s ingestion pipeline wrote one file per row — millions of files, each only a few kilobytes. These microfiles result in drastically slow reads, increased metadata overhead, and make optimization more expensive.

Lesson: Adjust partitioning or writer settings in your ingestion pipelines to produce fewer, larger files. Regularly compact small files using the OPTIMIZE command, or rely on Predictive Optimization to automate compaction for managed Delta tables. Excessive small files can significantly degrade query performance and increase compute and metadata management costs.

6. Large inserts and replaces drive up cost

Many teams refresh large datasets using INSERT OVERWRITE, REPLACE WHERE, or even CREATE OR REPLACE TABLE — for example, rewriting an entire 30-day time series when only a small subset of rows has changed. This results in billions of rows being rewritten unnecessarily each day.

Lesson: Replace full overwrites with incremental MERGE operations whenever possible. Optimize merge performance using Liquid Clustering. In one environment, this simple change reduced warehouse runtime by about 50%.

7. Cross-system queries often run slower without Photon acceleration

Federated queries — for example, Databricks reading from Snowflake or other external systems — often fall out of Photon execution mode, forcing execution to fall back to slower row-based processing. This can significantly increase query latency and compute cost.

Lesson: Keep frequently joined or high-volume tables within Databricks when possible, or create materialized views over foreign tables to reduce repeated cross-system access. Use Lakehouse Federation selectively to balance interoperability with performance. When an external system supports Delta Sharing for Delta Lake tables, it is typically a better option than federated queries because it preserves Photon acceleration.

8. Auto-stop settings are often too conservative

Many warehouses retain the default 10-minute idle timeout, or even increase it, despite intermittent workloads that provide frequent opportunities for the warehouse to scale down to zero. This leads to clusters sitting idle while still incurring DBU costs.

Lesson: Reducing the auto-stop setting to 5 minutes via the UI — or even 1 minute via REST API — can save thousands of dollars monthly, especially for scheduled or bursty workloads. For continuously running environments, consider right-sizing instead of keeping large warehouses idling.

9. BI tools can create inefficiencies by keeping connections alive and not properly cancelling queries

A frequent source of hidden inefficiency comes from BI tools that keep connections alive or fail to cancel submitted queries properly. In the first case, clients send periodic heartbeat queries to keep sessions open, preventing the warehouse from shutting down. In the second, clients stop polling for results from submitted queries, leaving the warehouse in a waiting state until it cancels them with the familiar “Query has been timed out due to inactivity” message.

Lesson: Review how your BI tools handle connection pooling, query cancellation, and session timeouts. Proper client configuration prevents idle workloads from consuming compute and improves overall warehouse efficiency.

10. Consolidating compatible workloads improves efficiency and reduces cost

Most teams maintain multiple warehouses for BI, ETL, ad hoc querying, metadata exploration, and data science. While such separation is necessary for workloads with different latency or concurrency requirements, running too many similar warehouses creates unnecessary startup, scaling, and caching overhead.

Lesson: Consolidate workloads with similar characteristics and latency expectations onto shared warehouses with appropriate scaling policies. This approach can deliver 20–40% cost savings, improves cache reuse, and simplifies governance. Use a small dedicated warehouse for lightweight metadata exploration instead of a large production warehouse.

Bonus Insights

• Avoid running OPTIMIZE after every write. In one case, a dbt job was configured to do exactly that — wasting DBUs without providing meaningful benefit. Let Predictive Optimization or a scheduled job handle compaction automatically.

• Larger warehouses don’t always mean better performance. A well-tuned Medium or Large warehouse can, in some cases, outperform an underutilized X-Large, delivering better efficiency at lower cost. Check query profiles to ensure queries parallelize effectively and make full use of available resources.

• Very high operator counts in SQL queries indicate data model complexity. Simplify by denormalizing or materializing gold tables to reduce query overhead and improve performance.

Conclusion

Across the Databricks SQL Cost & Performance Optimization Assessments, the most impactful improvements rarely came from sweeping architectural changes but from small, targeted adjustments — smarter auto-stop settings, better clustering, predictive optimization, or tuned queries. Collectively, these refinements translated into substantial cost reductions and faster, more predictable performance.

The key takeaway is simple: continuously measure, analyze, and iterate. Databricks SQL provides all the observability and automation needed to make optimization an ongoing practice, not a one-time project.

As a Specialist Solutions Architect at Databricks, I work with customers who want to compare Databricks SQL (DBSQL) performance against tools like Power BI, Tableau, and Databricks’ own AI/BI Dashboards. They’ll spin up a dashboard, hit “Refresh visuals,” time the result, and call that a benchmark.

Here’s the problem: production dashboards almost never run on a cold warehouse. Warehouses build up local disk cache as they serve repeated queries. If you benchmark against a cold cache, you’re not measuring what users actually see — you’re measuring the very first run after a restart.

This post introduces a production-friendly way to warm up the Databricks SQL disk cache using real historical queries. By replaying organic query patterns before testing, you can ensure your BI dashboards are benchmarked under realistic, steady-state conditions. The companion notebook provides all the code to automate this process and visualize when the cache has stabilized. You’ll be able to:

• Prime a warehouse’s disk cache using realistic dashboard traffic

• Keep benchmarks consistent across runs

• Avoid cheating by using the result cache

• Warm up one warehouse using the historical workload of a completely different warehouse

Why Benchmark with a Warm Cache

In production, BI dashboards rarely start from scratch. The Databricks SQL Disk Cache stores frequently accessed data locally on a warehouse cluster for faster retrieval. Ignoring this behavior during benchmarking can lead to unfair comparisons, especially if one test benefits from cached data while another does not.

A quick recap of caching layers in Databricks SQL:

• Cold run — nothing is cached yet; all data is read from storage.

• Warm run — data files are already cached on disk.

• Result cache — identical query results are returned instantly if underlying data hasn’t changed.

If you want to measure what end users actually experience in Power BI, Tableau, or Databricks’ AI/BI Dashboards, in most cases, you must benchmark against a warm cache, not a cold run.

The Warm-Up Approach

The companion notebook automates the warm-up process by extracting historical SELECT queries from the system.query.history table and replaying them with controlled concurrency. This effectively “primes” the disk cache with the same data your dashboards typically access.

Each warm-up session can be customized with parameters such as:

• Time range — which historical window of queries to extract

• User or service principal — whose queries represent dashboard workloads

• Warehouse ID — which warehouse’s query history to use as the source of truth

• Concurrency and delay — to simulate natural workload intensity

The notebook also tracks total execution times across multiple warm-up iterations and visualizes convergence — helping you see when the cache has stabilized and is ready for reliable benchmarking.

Extracting Historical Queries

The notebook begins by pulling real queries from system.query.history. You specify a user, warehouse, and time range that represents your normal dashboard activity:

query = f”“”
  SELECT statement_text
  FROM system.query.history
  WHERE start_time >= ‘{start_time_utc.isoformat()}’
    AND start_time <= ‘{end_time_utc.isoformat()}’
    AND executed_by = ‘{EXECUTED_BY}’
    AND compute.warehouse_id = ‘{WAREHOUSE_ID}’
  ORDER BY start_time
“”“

This ensures that the warm-up process reflects genuine production workloads, not synthetic benchmarks.

Avoiding Result Cache Shortcuts

If you replay the same queries exactly, Databricks SQL will return them from the result cache — defeating the purpose of warming the disk cache. To avoid that, the notebook automatically injects a harmless dynamic column into each query:

SELECT col_a, col_b, NOW() AS injected_now
FROM sales.fact_orders
WHERE ...

This simple trick forces each query to execute freshly while preserving the same scan patterns, ensuring that the disk cache — not the result cache — is exercised.

Alternatively, if you prefer not to modify queries programmatically, you can disable the result cache for the session before replaying queries:

SET use_cached_result = false;

This directive ensures that all queries execute freshly, regardless of whether identical results exist in the cache. For more details, see the Databricks documentation on query caching.

Replaying Queries with Concurrency

The modified queries are then replayed concurrently using the Databricks SQL Connector for Python. This concurrency simulates the burst of queries typical of a dashboard refresh:

from databricks import sql
import concurrent.futures, time

def execute_query(q):
    conn = sql.connect(**DBSQL_CONFIG)
    cursor = conn.cursor()
    cursor.execute(q)
    cursor.fetchall()
    cursor.close()
    conn.close()

def run_warmup(queries, passes, concurrency, delay_sec):
    durations = []
    for i in range(passes):
        start = time.time()
        with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
            pool.map(execute_query, queries)
        durations.append(round(time.time() - start, 2))
        time.sleep(delay_sec)
    return durations

In practice, you don’t need extreme concurrency to warm a single DBSQL warehouse. For most standard setups, 10 concurrent queries are sufficient to engage the warehouse’s local cache efficiently without overwhelming it. However, if you’re using multiple clusters per warehouse (for example, auto-scaled DBSQL configurations), you should scale concurrency proportionally — roughly 10 × N, where N is the number of clusters. This ensures that all clusters participate in the warm-up process and that data is cached across all of them.

Visualizing Cache Stabilization

After replay, the notebook visualizes warm-up durations across multiple passes to confirm that the cache has reached steady state:

import matplotlib.pyplot as plt

plt.figure(figsize=(8, 5))
plt.bar(range(1, len(durations)+1), durations)
plt.title(”Execution Time per Warm-up Pass”)
plt.xlabel(”Pass”)
plt.ylabel(”Duration (seconds)”)
plt.grid(axis=’y’)
plt.tight_layout()
plt.show()

You should see total duration drop sharply after the first run and then level off. When the bars flatten, the warehouse’s disk cache is effectively warmed, and your benchmark can begin.

The chart shows total warm-up duration across repeated passes for an X-Large DBSQL warehouse configured with 2 clusters (min = 2, max = 2). The first run is the slowest (cold cache). After several passes, execution time flattens — indicating the warehouse’s disk cache has converged and is ready for benchmarking.

Using Separate Warehouses for Query Extraction and Warm-Up

A powerful capability of this approach is that you can separate where you extract history from where you warm up.

This helps keep production environments safe and ensures that your benchmarks are isolated.

• Extraction warehouse: Connect to any SQL warehouse with access to system.query.history and filter for the production warehouse whose workload you want to reproduce. This step is lightweight and metadata-only.

• Warm-up warehouse: Use a different SQL warehouse (for example, a staging or benchmarking environment) to replay those queries and populate its disk cache.

This separation allows you to safely warm up a benchmarking warehouse using production-like workloads, without affecting your production dashboards or incurring additional load.

Why This Matters for BI Dashboards

Whether your users interact through Power BI, Tableau, or Databricks’ native AI/BI Dashboards, they all benefit from a warm disk cache. By simulating steady-state conditions before running benchmarks, you can:

• Ensure consistent, realistic performance testing

• Evaluate dashboard changes fairly

• Tune warehouse size and concurrency

• Compare tools and configurations on equal footing

Conclusion

To measure true user experience, always benchmark against a warmed Databricks SQL disk cache — not a cold one.

• Extract real queries from system.query.history

• Modify them to bypass the result cache

• Replay them with realistic concurrency (≈10 per cluster)

• Confirm warm-up stabilization

• Then test your dashboards

The companion notebook automates this entire workflow — including query extraction, replay, visualization, and optional warehouse separation — so you can focus on measuring performance, not preparing for it.

As data pipelines scale in size and complexity, keeping operational costs under control becomes increasingly important particularly for streaming workloads. Unlike batch jobs that run at scheduled intervals, streaming pipelines operate continuously, generating a steady stream of read and write operations. This constant interaction with cloud storage services like S3 can quickly accumulate API costs if not properly managed. Gaining visibility into these patterns and optimizing them is essential to maintaining efficient, cost-effective, and scalable data systems.

Why S3 API Calls Matter for Cost

S3 is widely used for its scalability, durability, and integration with modern data platforms. However, beyond storage fees, S3 also charges for API requests - an often underestimated factor that can significantly impact the cost of streaming data pipelines.

In autoloader or DLT workloads in Databricks, S3 is frequently accessed for operations such as file reads, writes, checkpoints, schema inference, and metadata management. These operations translate into S3 API calls, each of which incurs a cost depending on the request type.

Use case Overview:

As an example, let's take a Delta Live Tables (DLT) pipeline or a Structured Streaming job with the default trigger interval of 500 milliseconds to illustrate how S3 API calls can quietly drive up the cost of a streaming pipeline.

Assumptions with 500ms trigger interval:

• 100 S3 API calls every 500 ms = 200 API calls per second = 17,280,000 API calls per day
  

• Breakdown:

  • 40% PUT, LIST, POST → 69 calls/sec

  • 60% GET, READ → 131 calls/sec

📊 Daily API Call Volumes:

📅 Monthly Cost (30 Days):

$38.71/day × 30 days = ~$1,161.30/month

This is per pipeline cost and if we have 10 pipelines like this easily it can cost you ~10000/ month

Common Scenarios Leading to High S3 API Costs

Certain patterns such as small file writes, frequent checkpointing, or excessive schema discovery can unintentionally amplify S3 API usage. Since S3 charges per request type, even optimizations at the infrastructure or code level (e.g., reducing unnecessary LIST calls or batching writes) can have a noticeable impact on cost.

Understanding where and why S3 API calls occur in a streaming pipeline is crucial. Without this insight, organizations may see ballooning storage-related costs that don't correlate directly with the amount of data stored or processed. Therefore, monitoring and minimizing unnecessary API calls is key to building cost-efficient streaming architectures. Below are proven strategies to slash API expenses while maintaining performance.

Core Optimization Strategies

Option 1: Increase Trigger Interval in Bronze and Silver Layers

Reducing the frequency of micro-batch execution helps lower the number of S3 API calls, especially in high-volume streaming jobs where each trigger performs multiple GET, PUT, and LIST operations on S3. This optimization can be possible only when the latency requirement is not too low(in subseconds) per microbatch.

**The following metrics are based on the assumption that increasing micro-batch size leads to fewer files being read and written, as each batch processes more data.

New Assumptions with 2-Second Interval

• Old Trigger Rate: every 500 ms → 2 triggers/sec

• New Trigger Rate: every 2 seconds → 1 triggers/sec (consider only fewer files are written and the read rate remains same but writes are lower)

• Reduction Factor: 2× fewer triggers

• New API call rate:

  • 100 API calls every 1 seconds = 100 API calls per second

  • = 8,640,000 API calls/day

📊 Daily API Call Breakdown (Same 40/60 Split):

📅 Monthly Cost (30 Days):

$19.35/day × 30 = ~$580.50/month

This shows 2x reduction in cost per pipeline which can add up to multiple pipelines and help in significant reduction in cost.

Option 2: Table Properties Settings – Use v2 Checkpointing

Delta Lake’s v2 checkpointing is an optimized format that improves how Delta tables manage transaction logs. Unlike the default checkpointing (v1), which may trigger additional reads of Parquet data to gather stats or metadata, v2 checkpointing stores those stats directly in the checkpoint files. This reduces the need to make additional S3 GET or LIST API calls—thereby lowering I/O overhead and S3 request costs.

Why It Matters for Streaming:

In streaming pipelines, frequent checkpointing is common (especially in the bronze/silver layers). Traditional checkpoints often involve multiple S3 metadata reads. v2 checkpointing reduces this footprint by minimizing how often the engine needs to fetch additional files, helping control the number of REST API calls made to S3.

Benefits:

• Reduces S3 GET/LIST API calls

• Speeds up streaming reads and commits

• Lowers cloud storage access costs

To enable v2 Checkpointing set the below property:

ALTER TABLE my_table

SET TBLPROPERTIES (

'delta.feature.v2Checkpoint' = 'supported',

'delta.checkpointPolicy' = 'v2'

);

Option 3: Delta Lake Metadata Management

Over time, Delta Lake’s transaction log can grow significantly—especially in high-frequency streaming jobs. In one observed case, 30 days of logs resulted in 150GB+ of metadata in the _delta_log directory. This metadata bloat increases s3 list and get api calls.

Fix: Reduce Metadata Retention Durations

Set shorter retention periods for transaction logs and deleted files to control metadata growth:

ALTER TABLE silver SET TBLPROPERTIES (

'delta.logRetentionDuration' = '7 days',

'delta.deletedFileRetentionDuration' = '3 days'

);

Benefits:

• 85% smaller _delta_log directories

• Reduced metadata scanning and S3 API usage

• Lowers cloud storage access costs

Option 4: Increase the Frequency of Manual Maintenance Jobs the source and staging tables

In a continuous streaming pipeline, smaller files tend to accumulate in the intermediate tables due to high-frequency writes. Since each micro-batch writes individual records to storage, this results in a large number of small files, a scenario that can heavily degrade query performance and increase the frequency of S3 API calls (e.g., LIST and GET), leading to higher storage and I/O costs. So, it’s essential to run OPTIMIZE regularly to consolidate small files and better organize the data. Enabling auto-optimize and auto-compaction settings in Delta Lake can help automatically reduce the number of small files, ensuring more efficient file management and improved query performance.

Benefits:

• Optimizes large data volumes by consolidating smaller files, reducing file fragmentation.

• Fewer S3 LIST and GET API calls due to fewer, larger files.

• Continuous and regular maintenance ensures tables do not degrade over time due to excessive fragmentation.

Option 5: Reduce the Number of Min Batches to Retain

In streaming jobs, the minBatchesToRetain setting controls how many of the most recent micro-batches are retained in memory for processing. By default, spark.sql.streaming.minBatchesToRetain is set to 100, meaning the system keeps the latest 100 batches in memory. Lowering this number can help reduce API calls—particularly to S3—thereby optimizing costs.

Smaller State to Manage = Fewer Metadata Reads

When fewer micro-batches are retained, the streaming engine manages less state during job execution and checkpointing. Delta Lake may avoid reading and re-validating as many past logs or files (via list and get calls to s3) when cleaning up or updating internal state.
Impact: Leads to fewer S3 reads during log replay, especially after restarts or recovery operations.

Reduced Dependency on Historical Checkpoints

With a lower minBatchesToRetain, Checkpointing may reference fewer previous batch files. It also reduces interactions with _delta_log/ and old checkpoint files stored on S3.
Impact: Reduces the need to fetch older transaction log entries or metadata files.

When to Use This Setting:

• For low-latency, high-frequency streaming jobs with frequent checkpointing.

• When historical state is not needed for auditing or recovery beyond a few batches.

• As part of a broader S3 cost control strategy, alongside more direct tactics like optimizing checkpoint frequency, batching, and file compaction.

Option 6: Tune Shuffle Partitions Based on Cluster Size

By default, Spark sets spark.sql.shuffle.partitions = 200, which defines how many partitions are created during shuffles (e.g., after joins, groupBy, aggregations). In environments with fewer total available CPU cores, this leads to unnecessary parallelism, resulting in:

• Idle or underutilized partitions

• More metadata overhead

• Higher S3 API calls, especially LIST calls, when reading partitioned data or transaction logs

The number of S3 LIST and GET calls during a shuffle or file scan phase often scales with:S3 API calls ≈ listCallsPerPartition × number_of_partitions

Reducing partition count directly lowers the number of directory listings and metadata requests needed during delta transaction log reads, query planning, partition pruning and OPTIMIZE/Z-ORDER operations

When to Use: Total available cores × 2 < 200

Additional Recommendation for monitoring costs:

To speed up feedback and reduce reliance on delayed billing dashboards, use Amazon Athena to directly query S3 server access logs. This allows for:

• Near real-time visibility into GET, PUT, LIST, and DELETE requests

• Faster iteration and validation of pipeline optimizations

Final Thoughts

Streaming pipelines are powerful—but they can get expensive fast if you’re not careful with how often you hit services like S3. A lot of the cost sneaks in through frequent API calls, especially when you're writing small files, constantly listing buckets, or triggering jobs too often.

The good news? Small tweaks like increasing trigger intervals, batching writes, and cutting back on unnecessary reads can go a long way in saving money and making your pipeline run more smoothly.

At the end of the day, it's all about being smart with how your pipeline talks to S3. A few simple changes now can lead to big cost savings down the line—without sacrificing performance.

Vectorized (Pandas) UDFs marry Spark’s scale with Pandas & NumPy’s speed by streaming Arrow column batches across the JVM ↔ Python boundary. They are 10-100× faster than classic Python UDFs, but they still have sharp edges—batch sizing, unsupported types, 2 GB limits, executor memory, SafeSpark jail-sandboxes, etc.

In the world of big data processing, Apache Spark stands as the preeminent framework for distributed computation. One of its most powerful features for Python users is the ability to create User-Defined Functions (UDFs). However, traditional Python UDFs often face significant performance limitations. Enter Pandas PyArrow UDFs: a revolutionary approach that combines the analytical capabilities of pandas with the efficiency of Apache Arrow to deliver exceptional performance in distributed environments.

The Evolution of Python UDFs in Spark

Traditional Python UDFs in Spark suffer from three fundamental limitations that impact performance:

1. Serialization overhead: Data must be serialized between JVM and Python processes using pickle, which is computationally expensive2.

2. Row-by-row processing: Functions operate on individual rows rather than batches, resulting in millions of function calls for large datasets4.

3. Lack of vectorization: Operations can't leverage the optimized C/Cython implementations in pandas and NumPy libraries4.

Pandas UDFs were introduced in Spark 2.3 to address these limitations, with significant improvements in Spark 3.0 and beyond. These vectorized UDFs use Apache Arrow to efficiently transfer data and pandas to process it in a vectorized manner, delivering performance increases of up to 100x compared to traditional UDFs3.

Apache Arrow: The Backbone of High-Performance Data Exchange

Apache Arrow is the critical technology that enables the exceptional performance of Pandas UDFs in Spark. As an open-source columnar in-memory data format, Arrow was specifically designed to facilitate efficient data exchange between different programming environments2. For Pandas UDFs, Arrow eliminates the costly serialization/deserialization overhead that plagues traditional Python UDFs when transferring data between JVM and Python processes5.

Arrow achieves this efficiency through its columnar memory layout, which stores data contiguously by column rather than by row. This approach provides numerous benefits for analytical workloads: better memory compression, improved CPU cache utilization, and support for SIMD (Single Instruction, Multiple Data) vector operations4. Most importantly, Arrow enables a "zero-copy" shared memory model where both JVM and Python processes can access the same data without duplicating it, dramatically reducing the cost of data transfer6.

When a Pandas UDF executes, Spark converts data to Arrow format, splits it into batches, transfers these batches to Python workers as Arrow structures, processes them using pandas, and then returns the results via the same efficient Arrow pathway5. This entire pipeline is optimized for high-throughput, parallel processing across a distributed cluster. The result is performance gains that can transform previously impractical Python processing into viable production workflows3.

The Data Flow Process

The process of executing a Pandas UDF involves several steps that highlight how data flows through the Spark execution environment:

1. Spark converts the data into Arrow format

2. The data is split into batches (configured by spark.sql.execution.arrow.maxRecordsPerBatch)

3. Arrow batches are transferred to Python workers

4. Python workers convert Arrow batches to pandas Series or DataFrames

5. The UDF function processes these pandas objects

6. Results are converted back to Arrow format

7. Arrow data is transferred back to Spark

8. Spark converts Arrow data back to its internal format

This entire process happens in parallel across the Spark cluster, leveraging the distributed nature of Spark while maintaining the efficiency of vectorized operations.

Pandas UDFs Defined

You define a pandas UDF by decorating a Python function with @pandas_udf and adding type hints for the input and output:

from typing import Iterator
import pandas as pd
from pyspark.sql.functions import pandas_udf       

@pandas_udf('long')
def pandas_plus_one(iterator: Iterator[pd.Series]) -> Iterator[pd.Series]:
    return map(lambda s: s + 1, iterator)

display(spark.range(10).select(pandas_plus_one("id")))

• The signature (pd.Series → pd.Series) tells Spark which UDF flavor to pick.

• Under the hood, Spark uses Apache Arrow for zero-copy (de)serialization.

• Vectorized: your code gets whole batches as pd.Series/pd.DataFrame, not single cells.

Different flavours of Pandas UDF

• Series to Series (pandas.Series -> pandas.Series): This pattern exists to provide a clear, Pythonic, and type-hinted way to define vectorized UDFs that transform one Spark column into another, operating row-by-row (conceptually, applied batch-wise). It directly replaces the need to explicitly specify the older SCALAR Pandas UDF type, making the function's intent (operating on a Series and returning a Series of the same size) self-evident from the type hints.

• Iterator of Series to Iterator of Series (Iterator[pandas.Series] -> Iterator[pandas.Series]): This pattern was introduced to offer more flexibility and optimization for Series-to-Series transformations. It allows processing data in batches (iterators) rather than loading the entire column partition at once. The why is twofold: 1) It helps manage memory usage for very large data partitions, and 2) It enables expensive state initialization (e.g., loading a model) to be done once per batch iterator, improving performance.

• Iterator of Multiple Series to Iterator of Series (Iterator[Tuple[pandas.Series, ...]] -> Iterator[pandas.Series]): This extends the previous pattern because many operations require logic based on multiple input columns simultaneously. This type hint signature allows users to define UDFs that take batches of multiple input Series, perform calculations using them together, and return a single output Series batch, offering the same memory and initialization benefits for multi-column logic.

• Series to Scalar (pandas.Series -> Any): This pattern, often used with groupBy().agg() or window functions, provides a type-hinted way to define aggregations. It takes a Pandas Series representing a group or partition and returns a single scalar value. The why is to replace the older GROUPED_AGG Pandas UDF type with a more standard Python type hint signature, making the aggregation intent clear.

• applyInPandas (on GroupedData): This function exists specifically to implement the "split-apply-combine" pattern on grouped data6. Why? It allows applying a custom Python function, operating on a full Pandas DataFrame for each group, to perform complex, group-specific transformations or aggregations that are difficult or inefficient with standard Spark functions26. It expects one Pandas DataFrame (representing a group) as input and requires a Pandas DataFrame as output, effectively transforming each group256. Note: It loads the entire group into memory, which can be demanding for large groups6.

• mapInPandas (on DataFrame): This function exists to apply a Python function to an iterator of Pandas DataFrames, where each DataFrame represents a batch of data from a DataFrame partition13. Why? It's designed for transformations on entire partitions or batches where the logic is complex, best expressed in Python/Pandas, and crucially, where the number of output rows per input batch might differ from the input batch size (e.g., filtering rows, or mapping one input row to multiple output rows like unpacking files)1456. Its iterator-based approach aids memory efficiency for large datasets16.

Arrow Batch Size Considerations

The batch size for Arrow transfers is controlled by spark.sql.execution.arrow.maxRecordsPerBatch (default is typically 10,000 records). This setting can be tuned for performance:

• Smaller batches: Reduce memory pressure but increase overhead

• Larger batches: Better performance but higher memory requirements

For scalar operations, one partition consists of multiple Arrow batches. For grouping operations, one partition (group) is one Arrow batch, which can lead to out-of-memory issues if a group is too large6.

⚠️ Gotchas / Limitations

• Memory pressure: Each Arrow batch (or entire group in grouped UDFs) is copied into the Python worker’s RAM; large groups or oversized batches can OOM an executor. For grouped operations, all data for a group must fit in memory6

• Arrow caps:

  • Max rows per record batch = 2,147,483,647.

  • Some complex / deeply nested types (ArrayType(TimestampType), very nested Map/Struct) need recent PyArrow versions—or are unsupported.

  • Memory constraints: Setup overhead on tiny data: Python-worker spin-up and Arrow serialisation can outweigh benefits for small inputs; native Spark or even a pickled UDF may be faster. If you small data, honestly you should not worry just use Arrow.

🚀 Best Practices & Performance Tips

• Prefer native Spark first → Pandas UDF second → pickled Python UDF last.

• Vectorise: Use pandas/NumPy column ops, not Python loops.

• Tune batch size:

  • Start with default 10 k rows.

  • Reduce to ease memory or raise to improve throughput—set via spark.sql.execution.arrow.maxRecordsPerBatch.

• Provision RAM: Remember processing is in Python space; size executors for worst-case batch or group.

• Resource cleanup: In iterator UDFs, wrap model loads / file handles in try … finally.

• Type hints everywhere: Clearer code, earlier failures, faster Arrow conversion.

• Timestamps: Keep data in UTC; rely on pandas time-series APIs for conversions.

Our newsletter is 100% free and always will be, but without your claps, comments, or shares, search engines may bury this post forever. A quick clap not only tells us this content resonates but also makes sure you (and everyone else) can find it again when it matters most.

Performance Hierarchy:

1. Native Spark Functions: Generally the fastest as they operate entirely within the optimized Spark engine46.

2. Pandas UDFs (Vectorized): Offer significant speedups over traditional Python UDFs by using Arrow and vectorized processing. Can be 10-100x faster1346.

3. Arrow-Optimized Python UDFs: Faster than pickled UDFs due to efficient Arrow transport, but still process row-by-row logically1456.

4. Traditional (Pickled) Python UDFs: Slowest due to serialization overhead and row-by-row execution1456.

Conclusion: The Future of Python Data Processing in Spark & Arrow

Pandas PyArrow UDFs represent a breakthrough in solving the long-standing performance challenges of Python processing in Apache Spark. By bridging the gap between Spark's distributed computing capabilities and the rich ecosystem of Python data science tools, these UDFs enable data scientists and engineers to write high-performance Python code that can scale to massive datasets.

With the continued evolution of Arrow-optimized UDFs in Spark 3.5 and beyond, the performance advantages will only increase. The integration between pandas and Spark continues to deepen, making it easier for Python users to leverage their existing skills in a distributed environment while maintaining high performance.

As the ecosystem evolves, we can expect further innovations in this space, potentially eliminating the remaining barriers between Python's ease of use and Spark's distributed computing power, ultimately making complex data processing both simple and efficient.

TL;DR – Stop pickling rows. Start vectorizing columns. And may your Arrow batches be ever‑contiguous.

Why Deletes Are Hard in Streaming

Streaming pipelines are traditionally append-only. Systems like Kafka or files as a source don’t natively support deletes. But in real-world applications like GDPR, customers expect their data to be deleted across all layers: Bronze, Silver, and Gold.

Delta Lake Capabilities for DELETE

Delta Lake supports DELETE, UPDATE, and MERGE on batch and streaming tables. This unlocks powerful patterns for Change Data Capture (CDC), Slowly Changing Dimensions (SCD), and data correction.

• Supports ACID transactions

• Deletes propagate only if you plan for it

Said another way, Delta is more flexible than Kafka and Kinesis as a streaming solution.

Streaming Table Design for Delete Propagation

For append-only sinks, here’s a common pattern:

Bronze (Append-Only Example)

CREATE STREAMING TABLE bronze ( ... )
TBLPROPERTIES (
  delta.appendOnly = true
);

Note: If delta.appendOnly is set to true, no DELETE operations are allowed on that table. This setting locks the table to only support appends. So either we set the above or make sure to handle ignore DML commits (skipChangeCommits) or handle DML Commits (readChangeFeed). Anything else can cause failures in production.

Bronze (with Deletes/Updates/Merge)

ignore_dml_df = spark.readStream.format("delta") \
    .option("skipChangeCommits", "true") \
    .table("bronze.table_with_deletes")


change_feed_df = spark.readStream.format("delta") \
  .option("readChangeFeed", "true") \
  .table("bronze.table_with_deletes")

updates = change_feed_df.filter("_change_type = 'update_postimage'")
deletes = change_feed_df.filter("_change_type = 'delete'")

Understanding Deletion Vectors

Deletion vectors store metadata about logically deleted rows without immediately rewriting data files. They reduce the overhead associated with frequent DELETE operations, significantly improving DELETE efficiency.

• Reduces file rewrites

• Enhances performance of DELETE operations

• Automatically used by Delta Lake to keep track of logically removed rows

When you want to physically remove these rows from the data files, use REORG TABLE ... APPLY (PURGE).

Importance of VACUUM and REORG

VACUUM

After running DELETE operations, Delta tables contain logically deleted data but physically retain this data until explicitly vacuumed. Without running VACUUM, storage costs continually grow.

• Default retention period for VACUUM is 7 days, after which data files are physically removed, optimizing storage.

• Regularly running VACUUM ensures compliance with GDPR requirements by physically purging deleted data within a defined timeframe.

REORG TABLE ... APPLY (PURGE)

REORG is a command for rewriting data files in a Delta table that contain rows marked by deletion vectors. With APPLY (PURGE), logically deleted rows are physically removed:

REORG TABLE <table_name> APPLY (PURGE);

• Deletion vectors indicate rows that have been logically removed. REORG with APPLY (PURGE) re-creates data files without those rows.

• This process permanently applies deletions, removing the need to store deletion vectors.

• Essential for GDPR compliance when you need to ensure no residual data in physical storage.

Running both VACUUM and REORG periodically keeps your storage usage optimized and meets strict data compliance regulations like GDPR.

🙌 Keep This Post Discoverable: Your Support Matters!

We’re a small, independent team creating in-depth technical content like this to help the data engineering community. We don’t rely on sponsorships or ads—just passion and practical experience.

If you found this blog valuable, please take a moment to clap, comment, or share it. Your engagement helps search engines like Google surface this content for others and ensures you can find it again when needed.

Without interaction, even the most helpful posts can disappear into the internet void. Let’s keep high-quality content alive and accessible. 🙏

Subscribe now

Now back to the blog

Choosing Between skipChangeCommits and readChangeFeed

When to Use skipChangeCommits

• You do not care about Deletes/Merges/Update and you only need to handle appends then skipChangeCommits meets your needs.

only_appends_df = spark.readStream.format("delta") \
    .option("skipChangeCommits", "true") \
    .table("bronze.table_with_deletes")

When to Use readChangeFeed

• You need to explicitly handle all changes including DELETE and UPDATE operations.

• You want detailed tracking of changes (e.g., _change_type: insert, update_preimage, update_postimage, delete).

change_feed_df = spark.readStream.format("delta") \
  .option("readChangeFeed", "true") \
  .table("bronze.table_with_deletes")

updates = change_feed_df.filter("_change_type = 'update_postimage'")
deletes = change_feed_df.filter("_change_type = 'delete'")

Use Case: GDPR Delete Propagation

GDPR compliance demands timely deletion:

DELETE FROM bronze WHERE email = 'user@domain.com';
DELETE FROM silver WHERE email = 'user@domain.com';

After deletes:

1. Execute REORG TABLE <table_name> APPLY (PURGE) to physically purge rows marked in deletion vectors.

2. Run periodic VACUUM to remove data files no longer referenced.

Row-Level Concurrency

Databricks provides row-level concurrency, ensuring multiple writes (updates, deletes, merges) can be performed without corrupting the data. Instead of locking entire partitions or tables, Databricks manages concurrent writes at the file or row level.

How it Works

• Write Conflicts: If multiple transactions try to update the same rows simultaneously, Delta Lake checks the conflict at the row level.

• Partition/File Boundaries: For partitioned tables, concurrency can also be handled by partition-level transactions if updates touch different partitions.

• Isolation Levels: Databricks uses Snapshot Isolation by default and can escalate to Write-Serializable isolation when merges or updates overlap.

Why it Matters?

• Prevents data corruption.

• Ensures high availability — you don’t need to pause all jobs for one update.

• Complex streaming and batch patterns can coexist safely.

Summary and Recommendations

✅ Best Practice:

• set delta.appendOnly = true on tables where you want to lock your source table from DML operations.

• Perform deletes explicitly at Bronze and Silver layers.

• Periodically execute REORG TABLE ... APPLY (PURGE) and VACUUM.

• Decide between skipChangeCommits and readChangeFeed based on downstream needs.

FAQ

1. If I enable predictive optimization, do I need to run REORG / PURGE myself?
   Yes — enabling predictive optimization does not automatically execute the physical reorganization and purging of deleted data. You still must explicitly run REORG TABLE … APPLY (PURGE)

2. Do we need to pause batch and streaming jobs during the delete window?

   If you are only doing streaming appends alongside a single DML operation (like MERGE, UPDATE, or DELETE), conflicts are minimal. However, concurrent DML operations might lead to conflicts, resolved at the partition level for partitioned tables or file level using row-level concurrency.

3. How are concurrent updates handled in Spark?

   Databricks employs row-level concurrency controls. Conflicts from concurrent updates are handled at the file or partition level to ensure transaction integrity.

4. If we have two streaming tables, one upstream and one downstream, and skipChangeCommits enabled for both, can we safely run an UPDATE on both?

   Yes. Since they're independent streams, updates can safely occur as long as you use skipChangeCommits or properly handle deletes using Change Data Feed.

5. How often should I run Delete, REORG and Vacuum?

   Once a week, is a more cost effective strategy than daily to meet compliance needs. Order matters so do a DELTE, REORG and VACUUM in this sequence.

6. What happens if I set Vacuum to a large number?

   You will end up paying for storing logically deleted files meaning a big S3 or ADLS bill which you could have avoided.

7. What should your retention policy be?

   The default VACUUM retention period is 7 days, but you can choose a longer duration based on your data recovery needs. For instance, if someone accidentally deletes data and you’ve already vacuumed beyond the 7-day threshold, recovery via time travel becomes impossible.

   Consider real-world team dynamics—teams in North America often take two-week vacations. In such cases, setting a retention policy of 21 days might be more appropriate to allow for delayed incident detection and recovery.

   Remember: once a file is vacuumed, you can no longer time travel to a point before its deletion. A longer retention period buys your team more time to investigate and restore if necessary.

References

https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-delta-live-tables-dlt

• Delta Lake VACUUM

• Delta Lake REORG

• Deletion Vectors

• Row-Level Concurrency

• Skip Change Commits

• Change Data Feed

Introduction

While discussing concurrent query execution in Databricks SQL (DBSQL) with customers, I noticed a common misconception that I’d like to demystify in this blog. Databricks recommends a cluster for every 10 concurrent queries. This often sounds too low for customers with hundreds of analysts running BI dashboards containing dozens of queries each. With only 10 concurrent queries per cluster, can DBSQL truly support high-concurrency workloads? The answer is yes, and here’s how.

Definitions

Let’s start by defining two important concepts:

• Concurrent Query Execution refers to the number of queries that can run simultaneously on a SQL warehouse. It is constrained by compute resources, workload complexity, and system-enforced concurrency limits.

• Query Throughput measures the total number of queries processed over a given period (e.g., queries per hour, per minute, or per second). It depends on query execution speed, warehouse size, and workload optimization.

For example, if an isolated BI dashboard with 20 queries takes one minute to refresh, the throughput is 20 queries per minute (QPM). When 20 queries are submitted simultaneously, only about 10 of them will start executing immediately, while the other 10 may wait in a queue. In this case, concurrent query execution is 10, but it can be increased dynamically through autoscaling.

Concurrency Limits

Databricks recommends a cluster for every 10 concurrent queries. The maximum number of queries in a queue for all SQL warehouse types is 1,000 [1].

At first glance, 10 concurrent queries per cluster may seem low, especially when compared to Online Transaction Processing (OLTP) workloads, where processing many thousands of queries per second (QPS) is common. However, OLTP queries are relatively simple—they retrieve a "needle in a haystack" using optimized indexes. In contrast, Online Analytical Processing (OLAP) queries scan, process, and aggregate large datasets, requiring more compute resources.

Concurrency throttling is not unique to Databricks SQL; Snowflake, Redshift, BigQuery, and other cloud data warehouses enforce similar concurrency limits to prevent system overload. Both OLTP and OLAP databases implement query queuing to manage workloads efficiently.

Scaling Concurrency

DBSQL dynamically provisions compute resources (clusters) to increase concurrency. When new capacity is added, queued queries are automatically routed to the new compute resources [1].

The minimum and maximum cluster limits for autoscaling are configurable. For example, for workloads requiring 20 to 50 concurrent queries, setting Min = 2 and Max = 5 helps minimize queuing delays. Additionally, increasing the SQL warehouse size often improves query execution speed, leading to higher throughput.

The figures below illustrate the impact of SQL warehouse scaling on BI dashboard execution in DBSQL. In the first figure, with autoscaling settings Min = 1 and Max = 2, execution starts on one cluster, and after some queries get queued, DBSQL quickly adds another cluster. This setup demonstrates how DBSQL dynamically provisions additional compute resources to handle increased concurrency. In the second figure, the warehouse with a fixed cluster count Min = 2 and Max = 2 demonstrates that execution takes place on two clusters from the start, which avoids query queueing completely.

Figure 1: BI Dashboard Execution on DBSQL Large Min = 1 Max = 2

Figure 2: BI Dashboard Execution on DBSQL Large Min = 2 Max = 2

It is worth noting that, for Pro and Classic SQL warehouses, scaling is not instant, as it takes time to spin up new clusters. For Serverless SQL warehouses, scaling is near-instantaneous as resources are allocated from a pre-provisioned pool, eliminating startup delays. Serverless also has an advantage of Intelligent Workload Management (IWM), which predicts resource demands and adjusts capacity accordingly. Over time, IWM learns to make smarter decisions about scaling, ensuring efficient workload distribution and minimized latency.

Query Throughput and Performance Benchmarks

Here are two fully audited benchmarks that are available in the public domain since 2021. As of today, internal performance benchmarks indicate even higher throughput, though specific numbers cannot be disclosed.

Benchmark 1: Small-Scale Workloads

In 2021, DBSQL processed 14,777 TPC-DS queries per hour (QPH) on a 10GB dataset using a DBSQL Large warehouse without autoscaling [2].

To put this into perspective, if each BI dashboard refresh consists of 20 queries, then 14,777 / 20 = 738 dashboards can be refreshed per hour. In real-world scenarios, many of these dashboards and queries are likely to be served from the results cache, further increasing throughput and reducing compute resource consumption.

This demonstrates DBSQL’s efficiency for high-concurrency analytical workloads over small datasets, which are common in multi-tenant environments.

Benchmark 2: Large-Scale Workloads

In 2021, Databricks set an official world record by processing 32,941,245 queries per hour on a 100TB dataset using a DBSQL 4X-Large warehouse without autoscaling [3].

Following the same example, if each dashboard refresh consists of 20 queries, then 32,941,245 / 20 = 1,647,062 dashboards can be refreshed per hour.

The QphDS metric, used in the TPC-DS benchmark, represents the performance of mixed workloads. This essentially translates to nearly 33 million queries per hour, making DBSQL one of the most efficient solutions for large-scale analytical workloads.

Conclusion

By understanding and leveraging DBSQL’s concurrency management and scaling capabilities, organizations can efficiently support high-concurrency, high-throughput workloads. Properly sizing SQL warehouses, configuring autoscaling, and optimizing query execution strategies ensure timely and reliable analytical query processing, even for large-scale workloads. Whether handling dozens or millions of queries per hour, DBSQL provides one of the most efficient and scalable solutions available today.

References

[1] SQL Warehouse Sizing, Scaling, and Queuing Behavior

[2] New performance improvements in Databricks SQL

[3] Databricks Sets Official Data Warehousing Performance Record