Connect multiple Databricks Genie spaces to Slack with per-user authentication, intelligent routing, semantic caching, and audit logging — all deployed as a Databricks App. A production-grade Slack bot that brings Databricks Genie directly into Slack, allowing users to ask natural-language questions across multiple data domains without leaving their workspace.

Special Thanks

I extend my sincere thanks to Sunil Patil (https://www.linkedin.com/in/pppsunil/) for his invaluable help during the brainstorming phase. His insights were instrumental in solving the complex challenge of managing multiple genies across multiple Slack channels. This project definitely levelled up thanks to his help!

Features

• Multi-Genie Support: Route questions to the right Genie space based on channel, thread, AI classification

• Per-User Authentication: Use OAuth 2.0 User-to-Machine with PKCE so queries run under each user’s Unity Catalog permissions

• Threaded Conversations: Maintain Genie context across Slack threads

• Semantic Cache: Reuse answers for similar questions using Databricks Vector Search

• Delta-Aware Cache Invalidation: Automatically invalidate cached answers when underlying Delta tables change

• Audit Logging: Track user activity, cache hits, latency, errors, and Genie usage in Delta tables

• Reliable Genie Calls: Handle rate limits with retry, exponential backoff, and adaptive polling

• Native Deployment: Run the entire application as a Databricks App with no external infrastructure

Architecture

The application runs as a Databricks App and uses two services:

• Slack Bolt in Socket Mode to listen for messages, mentions, button clicks, and threaded replies

• Flask to handle OAuth callbacks and health checks

When a user asks a question in Slack, the bot follows this flow:

1. Checks whether the Slack user has a valid Databricks OBO token

2. Routes the question to the correct Genie space

3. Looks for a semantically similar cached response

4. Validates cached answers against Delta table versions

5. Sends the question to Genie on cache miss

6. Formats the response in Slack with SQL, tables, and suggested follow-ups

7. Writes cache metadata and audit logs to Delta

Prerequisites

1. Databricks workspace

• Active Databricks workspace

• One or more Genie spaces

• SQL Warehouse

• Unity Catalog permissions

• Vector Search endpoint and index

2. Slack App

• Slack workspace with admin access

• Slack app with Socket Mode enabled

• Bot token and app-level token

3. OAuth App

• Databricks OAuth application configured for User-to-Machine authentication

• Redirect URI pointing to the Databricks App callback endpoint

4. Download the application code and make the necessary updates to the config values in app.yaml. Download all files from repo: https://github.com/adgitdemo/ad_databricks/tree/main/multi-genie-multi-slack-channel-app

Setup Instructions

1. Slack App Setup

• Create a Slack app and enable Socket Mode.

• Under OAuth & Permissions, add these bot token scopes:

  • app_mentions:read

  • chat:write

  • im:history

  • im:read

  • im:write

  • channels:history

  • groups:history

  • users:read

• Subscribe to these bot events:

  • app_mention

  • message.im

  • message.channels

  • message.groups

• Install the app to your workspace and save:

  • Bot User OAuth Token

  • App-Level Token

  • Signing Secret

2. Databricks OAuth Setup

• Create a Databricks OAuth application for User-to-Machine authentication.

• Configure the redirect URI to point to the Flask callback route exposed by the Databricks App.

The bot uses OAuth 2.0 with PKCE to securely link each Slack user to their Databricks identity. After authentication, the user’s pending question is automatically processed, so they do not need to re-type it.

3. Databricks Workspace Setup

• Prepare the required Databricks resources:

  • Genie spaces

  • SQL Warehouse

  • Cache Delta table

  • Audit Delta table

  • Vector Search endpoint

  • Vector Search index

  • Catalog and schema permissions

• Permission Grants Summary

4. Configure the Application

• Update the application configuration with:

  • Slack bot token

  • Slack app token

  • Slack signing secret

  • Databricks workspace URL

  • OAuth client ID

  • OAuth redirect URI

  • Genie space mappings

  • Default Genie space alias

  • SQL Warehouse ID

  • Cache table name

  • Audit table name

  • Vector Search endpoint and index

  • Cache similarity threshold

  • Optional Service Principal fallback flag

• Example routing configuration:

{

“trips-data”: {

“space_id”: “your-trips-genie-space-id”,

“description”: “Questions about trips, pickup locations, fares, and distance”

},

“finance”: {

“space_id”: “your-finance-genie-space-id”,

“description”: “Questions about revenue, spend, forecasts, and financial KPIs”

}

}

Usage

In Slack

Direct Messages:

Send a DM to your bot

Ask questions like: “Show me sales data for last month”

Channel Mentions:

Invite the bot to a channel: /invite @YourBotName

Mention the bot: @YourBotName what are the top 10 customers?

Threaded Conversations:

Continue asking questions in a thread

The bot maintains conversation context within threads

Demo Time

In the General Slack Channel, the bot will use the default Genie Space to answer questions.

DM Thread, the bot will use the default Genie Space to answer questions.

In a dedicated Trip Slack Channel, the bot will use the Trip Genie Space to answer questions.

In a dedicated weather Slack Channel, the bot will use the Weather Genie Space to answer questions.

Real-time streaming in the Spark ecosystem (RTM) is a great leap in Spark structured streaming, as it guarantees an end-to-end processing as low as 5 ms

This directly serves use cases where ultra-low latency is crucial for operations, such as financial transactions and infrastructure monitoring, to name a few, which can directly translate to revenue loss or business image if not addressed in timely manners with Databricks’ RTM streaming

In this blog, we will introduce a simple demo notebook(s) focusing on:

• Creating a simulated streaming dataset in a message bus system (Kafka) with control over

  • Programmatically introducing duplicated records in the upstream

  • Queue partitions (As it impacts downstream processing/ patallelism)

  • Rows/Sec for writing into the message bus (Kafka is for demo purposes here but this can be any of the currently supported sources by RTM

• Reading real-time messages from the message bus in Spark real-time streaming mode and writing down to Databricks’ Lakebase serverless instance, including:

  • A simple, direct write (no transformation) using RTM

  • A simple stateful transformation (deduplication)

  • Explicit tuning parameters for both scenarios/use cases in order to quickly evaluate your use case for RTM

How to use this blog/code:

• Real-time mode sizing framework

  • You can change the upstream spec or

  • Downstream specs through the widgets that expose several tuning parameters

• Learning the ropes of a full pipeline, using foreach in a real-time streaming fashion to write down to a JDBC sink (Lakebase Postgres in our demo)

• Clone the code from the GitHub repo to start your journey!

Compute resources used:

• Any supported upstream system by Spark Realtime Mode

  • In this case, we are using a Kafka topic that we can recreate with n partitions (default 8)

• Databricks classic cluster (no autoscaling)

  • Driver: rd-fleet.xlarge (32 GB memory, 4 cores)

  • Workers: 8 rd-fleet.xlarge instances

  • DBR 17.3 LTS+

• Lakebase postgres instance with 2CU’s (Capacity Units)

The flow Illustration:

Important parameters:

In this framework, we included different parameters to control

Records generation (Upstream):

• Duplication of the same record (including the key) across different timestamps, to simulate a duplication happening upstream

• Percentage of duplicate records generated

• Partitions to be created (this will control the slots needed downstream when Spark reads from Kafka upstream in real-time fashion)

Records persistence (Downstream):

• Dedup flag in order to decide a stateful or stateless transformation

• A batching mechanism to increase throughput

Key Code Snippets to use/reuse:

I think the major part that I’d like to expand on is two things, actually:

• defining a Python JDBC-backed code to implement a Foreachwriter

  The structure of this implementation requires at least open, process, and close implementations. Additionally, I added a buffer implementation that is time or row counts based in order to control the throughput
  An implementation skeleton in Python would be as follows (full code available on GitHub)
  

  class PgForeachWriter:
      def open(self, partition_id, epoch_id):
          try:
              self.conn = psycopg2.connect(**conn_kwargs)
              self.conn.autocommit = False
              self.cursor = self.conn.cursor()
              
              # ...
  
              # Buffering state (only used in buffered mode)
              if use_buffered_mode:
                  self.buffer = []
                  self.last_flush_ts = time.time()
              
              return True
          
          except Exception as e:
              # ...
              return False
  
      def _flush_if_needed(self, force=False):
          ...
  
      def process(self, row):
          ...
  
      def close(self, error):
          ...
  
  
  def make_pg_buffered_writer(
      jdbc_url,
      jdbc_user,
      jdbc_password,
      jdbc_driver,
      jdbc_jar_path,
      table_name,
      max_batch_size=100,  # flush when >= this many rows (0 = simple mode)
      flush_secs=2.0,      # flush if last flush older than this (0 = simple mode)
  ):

• After defining the writer class, all you need is to use it in your writeStream with real-time mode, as shown below

  jdbc_writer = make_pg_buffered_writer(
      jdbc_url=JDBC_URL,
      jdbc_user=JDBC_USER,
      jdbc_password=JDBC_PASSWORD,
      jdbc_driver=JDBC_DRIVER,
      table_name=TABLE_NAME,
      max_batch_size=max_batch_size_param,
      flush_secs=flush_secs_param
  )
  
  query = (
      df_for_write
      .writeStream
      .foreach(jdbc_writer)
      .outputMode("update")
      .queryName("jdbc_sink_writer")
      .trigger(realTime="5 minutes")
      .option("checkpointLocation", checkpoint_path)
      .start()
  )

Performance measures:

I ran many scenarios in simulation and consistently sustained around 10k writes/sec over several minutes, with minimal to no delays and no backlog, as shown below. This aligns with a production rate of around 10K upstream writes, which are read in real time by Spark's streaming and written to Lakebase. I was also able to achieve a similar throughput for stateful (deduplication) transformation with no drops

Key considerations:

• RTM mode will handle any recs/sec rate when sized right

• Lakebase will handle writes/sec in the 10’s K range per CU out of the box

• That said, there are tuning parameters and things to consider:

  • To enhance the write throughput, and depending on the use case you can:

    • Consider Lakebase Autoscaling

    • Consider batching the writes to downstream (this is generally best practice) to benefit from Postgres batch DML API’s

    • Batching can be time or number-of-rows-based (example provided in the repo)

  • To enhance the RTM read in general:

    • You can tune your windowing range for stateful transformation

    • Tune the realTime value (default 5 minutes); this parameter is used for checkpoint commit

    • Increase the number of upstream partitions

The sweet spot:

• For the ideal optimum solution:

  • Consider right-sizing of:

    • upstream source and its partitions, as it will control parallelism downstream

    • Spark cluster size makes sure there are enough slots to schedule all stages at once

    • Downstream sizing (in our case, it is Lakebase) and consider autoscaling, more CU’s = More throughput

Conclusion:

While the purpose of this exercise is to demonstrate a full pipeline using Lakebase PostgreSQL as the downstream, and is not benchmarking specifically

The E2E latency in test is P95 356 MS with a P50 of 225 MS

These figures are out of the box with no tuning of any sorts, it can be improved further.

Also, with 8 Kafka topic partitions and an adequate Databricks cluster configuration, there is ~0 lag in real-time streaming read from Kafka and downstream write to Lakebase

That said, please note that your mileage will vary! In essence, that:

• A clear understanding of records production upstream is key

• A correct sizing of the Spark cluster and the Downstream OLTP (Lakebase Postgres in our setup) is crucial for throughput

• There is a trade-off between write capacity and the downstream cluster capacity units

  • Time or number of rows is a good tuning parameter in such a case Or

  • Autoscaling downstream to scale to more throughput requirements should be considered

🙋 Frequently Asked Questions

What is the main goal of this demo?

It demonstrates how to build a low-latency streaming pipeline from Kafka through Spark Real-Time Mode into Lakebase, while exposing the main tuning knobs that affect throughput and lag.

What variables can I change in the simulation?

You can adjust ingest rate, Kafka partition count, duplicate-record percentage, deduplication behavior, and downstream batching settings.

Why does Kafka partition count matter?

Partition count affects how much parallelism Spark can use when reading the stream. Too few partitions can limit throughput even if the cluster has spare capacity.

When should I use buffered writes instead of direct writes?

Buffered writes are usually better when downstream write throughput is the bottleneck. They reduce per-row overhead and improve efficiency for OLTP-style sinks.

Can stateful deduplication still perform well in Real-Time Mode?

Yes — based on our test results, the pipeline achieved similar throughput with deduplication enabled, provided the job was sized appropriately.

What should I tune first in production?

Start with upstream partitions, Spark cluster sizing, downstream write capacity, and batching thresholds. Then refine checkpoint cadence and stateful window settings.

As a Specialist Solutions Architect at Databricks, I often hear the same questions from customers who are migrating dashboards from legacy BI tools to Databricks AI/BI Dashboards:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

In the first blog post in this series, I focused on the first two questions and showed how to recreate:

• context filters using parameters in dataset SQL, and

• “Only Relevant Values” filters using field filters and query-based parameters.

In the second post, I focused on cross-filtering and drill-through interactions.

This third post tackles the remaining question: “How do we do user-based filtering in AI/BI Dashboards?”

In Databricks, these controls live in Unity Catalog, not in the dashboard itself. AI/BI Dashboards query governed tables and views, and Unity Catalog enforces fine-grained access control before the data ever reaches the dashboard. I’ll walk through how to:

• Implement user-based filtering and RLS with dynamic views that use current_user() and is_account_group_member().

• Apply RLS directly on tables using row filters and protect sensitive fields with column masks.

• Scale these patterns across many tables and columns using ABAC tag policies and governed tags.

As in the previous posts, I’ll use the built-in samples.tpch dataset. I’ve also published the companion dashboard so you can import it into your workspace, follow along as you read, and adapt these patterns to your own Unity Catalog data.

1. How User-Based Filtering Maps to Unity Catalog

Before we dive into SQL, it’s useful to clarify where these responsibilities live in Databricks.

In many BI tools, user-specific security is often implemented close to the dashboard or semantic layer:

• You define user- or group-based rules that map principals to specific regions, customers, or business units.

• You may use identity-aware logic in filters or calculated fields.

• You may maintain a security table that drives which slice of data each user can see.

In Databricks, these controls live in Unity Catalog, not in AI/BI Dashboards:

• Object privileges on catalogs, schemas, tables, and views control whether a user can query a given object at all.

• Dynamic views, row filters, and column masks implement row-level security and masking at query time. They can inspect the current user and their groups and return different rows or values per user.

• AI/BI Dashboards simply query those governed tables and views. They never bypass Unity Catalog: any row filters or masks you define apply to every query, regardless of whether it comes from a notebook, Databricks SQL, or an AI/BI dashboard.

The result is conceptually similar to user-based filtering in traditional BI tools, but with one important shift: The security rules live with the data, not with a particular dashboard.

That’s especially important when:

• The same Unity Catalog tables power multiple AI/BI dashboards and external BI tools, and

• You embed AI/BI Dashboards into applications where thousands of users see the same dashboard definition, but each must see a different subset of data.

In the rest of this post, we’ll build up from that idea:

1. Use a dynamic view and a permission table to enforce RLS on a TPCH Sales dataset.

2. Show how to do similar things directly on tables with row filters and column masks.

3. Discuss how to scale those patterns across many tables and columns using ABAC tag policies and governed tags.

2. Building blocks: dynamic views, row filters, and column masks

To implement user-based filtering in Databricks, you really only need three Unity Catalog primitives: dynamic views, row filters, and column masks.

They all rely on the same core idea: At query time, Unity Catalog can look at who is running the query (and which groups they’re in), and then decide which rows and values to return.

2.1 Identity functions

The main functions you’ll use in policies are:

• current_user()
  Returns the current user’s identity (usually their email).

• is_account_group_member(’<group_name>’)
  Returns TRUE if the current user is a member of an account-level group.

You can call these functions from views and from SQL UDFs used by row filters and column masks.

2.2 Dynamic views

A dynamic view is just a normal SQL view whose logic depends on the current user or their groups.

You can:

• Filter rows based on current_user() or group membership.

• Mask or null out columns for certain users.

• Join to a separate permission table that maps users/groups to allowed regions, customers, etc.

Any AI/BI dataset that selects from a dynamic view automatically inherits its logic. You don’t need to add any special configuration in AI/BI itself.

We’ll use a dynamic view for our first TPCH Sales RLS example.

2.3 Row filters

A row filter attaches RLS logic directly to a table, instead of wrapping the table in a view:

• You define a SQL UDF that takes one or more columns as input and returns BOOLEAN.

• You attach it to a table with ALTER TABLE ... SET ROW FILTER ... ON (column[, ...]).

The row filter runs for every query and can call current_user() and is_account_group_member() internally. This is handy when you want:

• A stable table name (no extra view layer), or

• A single table that’s consumed by many tools, all of which should respect the same RLS.

We’ll look at both group-based and current_user() + permission-table examples later in the post.

2.4 Column masks

A column mask is similar, but operates at the column level:

• You define a SQL UDF that returns a “masked” value.

• You attach it with ALTER TABLE ... ALTER COLUMN ... SET MASK ....

This lets you:

• Show full values (for example, email or salary) only to certain groups.

• Show partially masked or null values to everyone else.

Think of it as the Unity Catalog side of “row-level security + column-level masking” that you might combine in legacy BI tools using data source filters and calculated fields.

Next, we’ll put these pieces together in a concrete example: enforcing region-based RLS on a TPCH Sales dataset using a combination of a base view, a permission table, and a dynamic view.

3. Implementing RLS with a dynamic view

Let’s start with a concrete scenario:

• You have a TPCH Sales dashboard shared across multiple sales teams.

• NA Sales Managers should see only AMERICA.

• EMEA Sales Managers should see only EUROPE.

• APAC Sales Managers should see only ASIA.

• Individual users may have their own custom regions.

In many traditional BI tools, you’d typically solve this with a user filter or data source filter that maps groups to Regions, and a security table to keep that mapping up to date.

In Databricks, we’ll use the same logical pattern – but move it into Unity Catalog:

1. Create a base view for TPCH Sales in a demo schema.

2. Create a permission table that maps principals (groups or users) to Regions.

3. Create a dynamic view that joins the base view to the permission table and applies row-level security based on current_user() and is_account_group_member().

4. Create an AI/BI dataset on top of that dynamic view and build a simple table visualization.

Throughout this section, we’ll:

• Read from samples.tpch (which everyone has).

• Create objects in main.demo_tpch (you can substitute another catalog/schema if needed).

All of the CREATE / INSERT statements in this section should be run in a notebook or SQL editor, not inside an AI/BI dataset. In AI/BI, you’ll just SELECT from the resulting view.

3.1 TPCH Sales base view

First, set up a simple demo schema and define a reusable base view for TPCH Sales:

-- Use the default UC catalog and create a demo schema
USE CATALOG main;
CREATE SCHEMA IF NOT EXISTS demo_tpch;

-- Base TPCH Sales view, reading from samples.tpch
CREATE OR REPLACE VIEW main.demo_tpch.tpch_sales_base AS
SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

3.2 Region access permission table

Next, create a permission table that describes who is allowed to see which Region.

-- Optional: you can keep security tables in the same schema
-- or create a separate one, e.g. main.demo_security
CREATE SCHEMA IF NOT EXISTS main.demo_tpch;

CREATE TABLE IF NOT EXISTS main.demo_tpch.tpch_region_access (
  principal_type STRING,   -- 'group' or 'user'
  principal      STRING,   -- group name or user email
  region         STRING    -- must match tpch_sales_base.region
);

INSERT INTO main.demo_tpch.tpch_region_access VALUES
  ('group', 'NA Sales Managers',        'AMERICA'),
  ('group', 'EMEA Sales Managers',      'EUROPE'),
  ('group', 'APAC Sales Managers',      'ASIA'),
  ('user',  'some.user@databricks.com', 'ASIA'),
  ('user',   current_user(),            'AFRICA');

What this does:

• The first three rows grant access based on account-level groups

• The fourth row grants access to ASIA to a specific user, even if they are not in one of those groups.

• The fifth row uses current_user() to grant you, the person running this SQL, access to AFRICA. When you execute the INSERT, Unity Catalog evaluates current_user() to your own email.

If you’re not in any of the NA/EMEA/APAC Sales Managers groups and you’re not some.user@databricks.com, the only applicable rule for you will be the one that says you can see AFRICA. We’ll see the effect of that in a moment when we query through the dynamic view.

3.3 Dynamic view with region-level RLS

Now create a dynamic view that applies row-level security by joining the base TPCH Sales view to the permission table and checking the current user’s identity and groups:

CREATE OR REPLACE VIEW main.demo_tpch.tpch_sales_rls AS
SELECT s.*
FROM   main.demo_tpch.tpch_sales_base AS s
WHERE EXISTS (
  SELECT 1
  FROM   main.demo_tpch.tpch_region_access a
  WHERE  a.region = s.region
    AND (
      (a.principal_type = 'group'
       AND is_account_group_member(a.principal))
      OR
      (a.principal_type = 'user'
       AND a.principal = current_user())
    )
);

This view enforces RLS as follows. For each row in tpch_sales_base, it looks for a matching rule in tpch_region_access based on region and either group or user principle. If no matching rule exists for the current user and that region, the row is filtered out.

You can verify this by running:

SELECT DISTINCT region
FROM main.demo_tpch.tpch_sales_rls;

If you are only granted access via the (’user’, current_user(), ‘AFRICA’) row, you should see: AFRICA.

From now on:

• Any query against main.demo_tpch.tpch_sales_rls returns only the regions granted to the current user.

• This applies uniformly whether the query comes from a notebook, Databricks SQL, or an AI/BI Dashboard.

• You can add or revoke access simply by inserting or deleting rows in main.demo_tpch.tpch_region_access – you don’t need to change the view logic.

3.4 Using the dynamic view in an AI/BI dataset

With main.demo_tpch.tpch_sales_rls in place, using it in AI/BI Dashboards is straightforward. You don’t need to re-implement any RLS logic in AI/BI – the dataset just selects from the governed view.

Create the TPCH Sales (RLS Dynamic View) dataset:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_rls;

Build a simple dashboard page to visualize the dataset. In the companion dashboard, I created a RLS with dynamic view page based on this dataset:

When you view this page, if your only grant is the row we inserted with (’user’, current_user(), ‘AFRICA’), the table will show only rows where the region is AFRICA.

The security logic lives in Unity Catalog (dynamic view + permission table). The dashboard just selects from tpch_sales_rls and automatically respects row-level security for each viewer.

4. Implementing RLS on tables with row filters and column masks

In the previous section, we implemented row-level security for TPCH Sales using a dynamic view and a permission table. That pattern works well when you want a named, shareable view to point AI/BI datasets at.

Unity Catalog also lets you attach RLS and masking logic directly to tables using:

• Row filters – control which rows a user can access in a table.

• Column masks – control what values they see in specific columns.

These policies are evaluated in Unity Catalog at query time and apply to all compute – SQL warehouses, notebooks, and AI/BI Dashboards. Unlike dynamic views, they keep the table name unchanged, which can be important when the same table is shared across many tools.

In this section, we’ll:

1. Create a TPCH Sales table in main.demo_tpch for row filters and masks.

2. Attach a group-based row filter that restricts Regions.

3. Attach a user-based row filter that uses current_user() and a permission table.

4. Add a column mask to protect a sensitive column.

Any AI/BI dataset that selects from this table will automatically respect these policies. You don’t need to configure anything special in AI/BI.

All of the statements below should be run in a notebook or SQL editor. AI/BI Dashboards just query the resulting table.

4.1 TPCH Sales table for filters and masks

Row filters and column masks only apply to tables (and a few other relation types), not to views. To keep things simple, we’ll materialize the tpch_sales_base view from Section 3 into a Delta table that we can attach policies to:

-- Use the same demo catalog and schema as before
USE CATALOG main;
USE SCHEMA demo_tpch;

-- Create a physical table from the base view for row filters and masks
CREATE OR REPLACE TABLE main.demo_tpch.tpch_sales_table AS
SELECT *
FROM main.demo_tpch.tpch_sales_base;

From now on, we’ll attach row filters and masks to main.demo_tpch.tpch_sales_table.
If you point an AI/BI dataset at this table instead of the dynamic view, the behavior will be controlled by these table-level policies.

4.2 Group-based row filter on Region

First, let’s attach a row filter that enforces the same Region rules we used in the dynamic view, but purely based on account-level groups:

-- Row filter function that decides which regions each group can see
CREATE OR REPLACE FUNCTION main.demo_tpch.tpch_region_filter(p_region STRING)
RETURNS BOOLEAN
RETURN
  CASE
    WHEN is_account_group_member('NA Sales Managers')   THEN p_region = 'AMERICA'
    WHEN is_account_group_member('EMEA Sales Managers') THEN p_region = 'EUROPE'
    WHEN is_account_group_member('APAC Sales Managers') THEN p_region = 'ASIA'
    ELSE FALSE
  END;

Attach it to the table:

ALTER TABLE main.demo_tpch.tpch_sales_table
  SET ROW FILTER main.demo_tpch.tpch_region_filter ON (region);

Effect:

• Whenever anyone queries main.demo_tpch.tpch_sales_table, Unity Catalog evaluates tpch_region_filter(region) for each row.

• If the user is in NA Sales Managers, only rows where the region is AMERICA are returned. If they’re in EMEA Sales Managers, they only see EUROPE; APAC Sales Managers see ASIA.

• Users not in any of these groups see no rows from this table.

If you point a dataset at:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_table;

Any visualizations based on such a dataset will now respect the group-based Region logic without going through the dynamic view.

4.3 User-based row filter with current_user() and a permission table

Group-based rules are great for broad roles, but you may need finer control – different users seeing different subsets of customers, accounts, or regions.

We can reuse the same pattern as in Section 3 – current_user() + a permission table – but this time embed it in a row filter function instead of a dynamic view.

Example: restrict access by customer_id:

-- Permission table mapping users to customers they can see
CREATE TABLE IF NOT EXISTS main.demo_tpch.customer_access (
  user_email  STRING,
  customer_id BIGINT
);

-- Example grants
INSERT INTO main.demo_tpch.customer_access VALUES
  ('some.user@databricks.com', 889),
  (current_user(),             1111);  -- Give yourself access

Now create a row filter function that consults this table. To avoid ambiguous name resolution with the customer_id column on the table, we’ll use a parameter name p_customer_id:

CREATE OR REPLACE FUNCTION main.demo_tpch.tpch_customer_filter(p_customer_id BIGINT)
RETURNS BOOLEAN
RETURN EXISTS (
  SELECT 1
  FROM   main.demo_tpch.customer_access a
  WHERE  a.user_email  = current_user()
    AND  a.customer_id = p_customer_id
);

Attach it to the same table. Because a table can have only one row filter, we’ll replace the Region filter from the previous subsection in this example:

ALTER TABLE main.demo_tpch.tpch_sales_table
  DROP ROW FILTER;

ALTER TABLE main.demo_tpch.tpch_sales_table
  SET ROW FILTER main.demo_tpch.tpch_customer_filter ON (customer_id);

Effect:

• Whenever someone queries main.demo_tpch.tpch_sales_table, Unity Catalog evaluates tpch_customer_filter(customer_id) for each row.

• For a given user, only rows whose customer_id appears in main.demo_tpch.customer_access
  for current_user() are returned.

• In the sample data above, you will only see orders for customer 1111, while some.user@databricks.com will see orders for customer 889.

You can verify this quickly:

SELECT DISTINCT customer_id
FROM main.demo_tpch.tpch_sales_table
ORDER BY customer_id
LIMIT 20;

If your only mapping is (current_user(), 1111), this query should return just 1111.

To use this in AI/BI Dashboards, you can point a dataset directly at the table:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_table;

In the companion dashboard, I created a RLS with row filter page based on this dataset:

When you open it, the table and visuals only show data for customers you are allowed to see according to customer_access, without any RLS logic in the dashboard itself.

4.4 Bonus: masking sensitive columns with a column mask

Row filters decide which rows a user can see. Sometimes you also need to partially hide sensitive values within those rows – for example, masking customer names, emails or phone numbers for most users while leaving them fully visible for a small group.

Unity Catalog column masks handle this at the column level using the same pattern: a SQL UDF that can branch on current_user() or group membership.

Suppose we want:

• Users in a PII Full Access group to see full customer names.

• Everyone else to see a partially masked version (for example, just the first few characters).

First, define a masking function:

CREATE OR REPLACE FUNCTION main.demo_tpch.mask_customer_name(name STRING)
RETURNS STRING
RETURN
  CASE
    WHEN is_account_group_member('PII Full Access') THEN name
    ELSE concat(substr(name, 1, 3), '***')
  END;

Attach it as a mask on the table:

ALTER TABLE main.demo_tpch.tpch_sales_table
  ALTER COLUMN customer_name
  SET MASK main.demo_tpch.mask_customer_name;

Effect:

• Users in the PII Full Access group see the full customer_name value.

• All other users see a masked version like Cus*** instead of Customer#000001111.

• The mask is enforced for every query against tpch_sales_table – notebooks, SQL editor, and AI/BI Dashboards – including exports.

Here is what it looks like in the companion dashboard:

5. Scaling RLS with ABAC and governed tags

Everything we’ve done so far (dynamic views, row filters, and column masks) is defined directly on individual Unity Catalog objects. That’s fine for a handful of tables, but it becomes hard to manage when you have dozens of catalogs, hundreds of schemas, and thousands of tables. This is exactly the problem that attribute-based access control (ABAC) with governed tags is designed to solve in Unity Catalog.

At a high level, ABAC adds three building blocks on top of the mechanisms we already used:

• Governed tags – account-level tags like sensitivity, business_domain, or region_scope, with a controlled set of allowed values. You attach these tags to catalogs, schemas, tables, or columns.

• Policy UDFs – reusable SQL UDFs that implement row-filter or column-mask logic, similar to the functions we wrote earlier, but intended to be reused across many datasets.

• ABAC policies – centrally managed policies that say “when a tagged object matches these conditions, apply this row filter or column mask for these principals.” Policies can be attached at the catalog, schema, or table level and inherit down the hierarchy.

Databricks recommends using ABAC as the primary way to apply row filters and column masks at scale, and reserving table-by-table configuration for special cases. Conceptually, you can think of ABAC as “lifting” the patterns from Section 4 into a central policy layer:

1. Tag the data once – apply governed tags to catalogs, schemas, tables, and columns that participate in RLS or masking.

2. Register reusable UDFs – define shared row-filter and mask functions in a governance schema (for example, governance.region_filter() and governance.mask_customer_name()).

3. Create ABAC policies ​​– define policies that attach those UDFs to tagged objects based on tag conditions and target groups.

4. Let tags drive behavior – as new tables and columns are tagged, the appropriate row filters and masks are applied automatically by Unity Catalog.

From an AI/BI Dashboards perspective, the experience is the same as in Sections 3 and 4: the dataset SQL stays simple, and the dashboard filters and visualizations work as usual. The difference is that the security logic is now centralized in ABAC policies and tags instead of being embedded directly into each table or view.

6. Summary and next steps

In many BI tools, user-based filtering and row-level security are often implemented close to the dashboard or semantic layer using user/group mappings, security tables, and identity-aware logic. In Databricks, the key shift is that these controls move into Unity Catalog, and AI/BI Dashboards simply query governed tables and views.

In this post, we walked through three main patterns:

• Dynamic views and permission tables (Section 3)
  We built main.demo_tpch.tpch_sales_rls on top of a base TPCH Sales view and a tpch_region_access table. The dynamic view uses current_user() and is_account_group_member() to return different Regions for different users and groups. AI/BI datasets that query this view automatically inherit the row-level security.

• Table-level row filters and column masks (Section 4)
  We materialized TPCH Sales into main.demo_tpch.tpch_sales_table and attached a row filter that looks up allowed customer_id values in customer_access based on current_user(). We also added a column mask for customer_name, showing full names only to a privileged group. Any dataset that selects from this table sees the combined effect of RLS and masking without any extra logic in the dashboard.

• ABAC and governed tags (Section 5)
  We then zoomed out to show how ABAC can apply the same kinds of row filters and masks at scale, using governed tags, reusable policy UDFs, and central ABAC policies. Instead of configuring each table or view by hand, you tag data once and let policies attach the right filters and masks automatically.

Across all three patterns, the core idea is the same: AI/BI Dashboards stay simple; Unity Catalog enforces who sees which rows and what values.

The companion dashboard brings these ideas together in a concrete, runnable example. If you import it into your workspace and wire it up to the views and tables from this post, you can see exactly how the visuals behave for different users as Unity Catalog applies dynamic views, row filters, and column masks behind the scenes.

Combined with the first two posts in this series (Part 1 and Part 2), you now have a practical set of patterns for implementing filtering, drill-through, and row-level security in Databricks AI/BI Dashboards on top of Unity Catalog.

• The Problem: A “Full Refresh” on Kafka-fed pipelines can cause permanent data loss if older records have aged out of the topic (TTL).

• The Strategy: Use a “Backup-and-Rebase” workflow: archive existing data, identify the last processed offsets, and point the new pipeline to that exact starting position.

• The Execution: This guide demonstrates how to manually configure startingOffsets in Spark to bridge the gap between historical backups and new Kafka data.

• The Result: Seamless streaming table updates with zero data loss or record duplication.

The Constraint: Kafka TTL & Streaming Table Immutability

In Spark Declarative Pipelines (SDP), situations may arise when you need to alter a bronze Streaming Table that is being fed by Apache Kafka. This can pose a challenge as you cannot manually alter Streaming Tables via Alter Table commands.

This is further complicated by the fact that Kafka topics are configured with a finite retention period (Time-to-live or TTL), meaning older records eventually age out. Since a pipeline Full Refresh clears the target Delta Table, you cannot simply update your pipeline definition and full refresh from the source, as older records will be missing. The diagram below illustrates this situation. The Kafka cluster contains user-4 who has already been ingested, and users 5 and 6 who still need to be ingested into our Lakehouse. The older user records (1, 2, and 3) have aged out of the topic’s TTL.

As new records are constantly being appended to this topic, how can I update my pipeline and alter my result table without missing new records (users 5, 6, etc), dropping old records (users 1, 2, and 3), or duplicating records (user-4)? The following example has been fabricated to show the solution. The actual reason for implementing this will vary by use-case.

Example: Ingesting JSON via Kafka

Imagine a pipeline is ingesting JSON data from Kafka. This JSON data contains three high-level fields: name, country, and email. It also contains two nested fields “event” and “device” which contain information about what actions users are taking from which devices.

{”user”:”user-2”,”email”:”user-2@example.com”,
“country”:”CA”,
“device”:{”os”:”android”,”model”:”Pixel 7”,”geo”:{”lat”:39.98,”lon”:-82.98}},
“event”:{”name”:”demo”,”seq”:2,”ts”:”2026-02-17T14:40:26.189Z”}}

This data is being written to a json_bronze table, and is storing the user, email, and country fields as String and the nested fields as Struct types. It also adds Kafka metadata fields.

import dlt
from pyspark.sql.functions import current_timestamp, col, from_json, expr

SERVERS = "REDACTED"
TOPIC = “neil_struct_topic”

# Explicit JSON schema for predictable demos (no schemaLocationKey)
SCHEMA_DDL = “”“
  user STRING,
  email STRING,
  country STRING,
  device STRUCT<os: STRING, model: STRING, geo: STRUCT<lat: DOUBLE, lon: DOUBLE>>,
  event STRUCT<name: STRING, seq: BIGINT, ts: STRING>
“”“

@dlt.table(
    name=”json_bronze”,
    comment=”Raw Kafka payload with explicit JSON schema and rescued data”,
    table_properties={”quality”: “bronze”}
)
def json_bronze():
    df = (
        spark.readStream
            .format(”kafka”)
            .option(”kafka.bootstrap.servers”, SERVERS)
            .option(”kafka.security.protocol”, “SSL”)
            .option(”subscribe”, TOPIC)
            .option(”startingOffsets”, “earliest”)
            .load()
    )

    parsed = (
        df.selectExpr(”CAST(value AS STRING) AS json_str”, “topic”, “partition”, “offset”, “timestamp”)
          .select(
              from_json(
                  col(”json_str”),
                  SCHEMA_DDL,
                  options={”rescuedDataColumn”: “_rescued_data”}  # capture type mismatches/new fields
              ).alias(”data”),
              “topic”, “partition”, “offset”, “timestamp”
          )
          # data.* includes _rescued_data already; do not reselect it to avoid duplicate column error
          .selectExpr(”data.*”, “topic”, “partition”, “offset”, “timestamp AS kafka_timestamp”)
          .withColumn(”ingestion_ts”, current_timestamp())
    )

    return parsed

Below is the current state of our target table. This matches the state of the diagram above. Users 1 through 4 have been ingested into the target Delta table.

Now let’s imagine the nature of our topic changes and the device and event fields need to become flexible, allowing for new nested fields to be added anytime and reflected in our target. In the current implementation, these new fields would not appear automatically in our Struct columns.

One way to allow for flexibility of these nested columns is to update the Struct columns to Variant type. As mentioned above, however, Alter Table is unavailable on a streaming table to update column types. Here’s how to accomplish a streaming table update while ensuring there is no data-loss or duplication.

Step 1: Pause your pipeline

Whether your pipeline is continuous or scheduled to run periodically, you don’t want to be ingesting data while performing these actions.

Under the scheduled Job:

Step 2: Backup Bronze Table

This retains all data we’ve already ingested (users 1-4), including data that no longer exists in our Kafka topic (users 1-3). We’ve now ensured we won’t lose records that have aged out of the source.

CREATE TABLE neil_test_catalog.streaming.json_bronze_backup
SELECT * FROM neil_test_catalog.streaming.json_bronze;

Step 3: Determine Max Offset per Partition

At this point, Kafka has continued to receive new records behind the scenes (user-5 and user-6). We want to ensure that when we Full Refresh our pipeline, we read only these new messages without reprocessing data (user-4). This is achieved by making use of Spark’s readStream startingOffsets parameter.

This parameter allows you to specify which offsets Spark should begin reading data from, the first time a pipeline runs. Keep in mind that in Kafka, offsets are integers that uniquely identify messages per partition, so you’ll have to specify a starting offset for each partition in your topic. SDP uses these offsets to ensure that upon initial startup, the pipeline begins exactly where you intend. From that point forward, Spark continuously records these offsets in its internal checkpoints to track progress over time and guarantee exactly-once processing. It’s also a good idea to store this Kafka metadata in the Delta table itself.

Here’s the same snapshot of our source topic and target Delta table, with offset information included:

Notice that to begin processing at user-5 we will need to specify our pipeline start at offset 4. The startingOffsets parameter expects this information in the following format:

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)

If our topic contained multiple partitions, starting offsets must be set for each partition and would look like this for two partitions:

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4, “1”:6}}’)

But how can we find this information?

Finding via Metadata Columns (If Defined in Pipeline and Tracked in Delta Table)

If you’ve added Kafka metadata to your bronze table, you can retrieve your max offset per partition there. Remember that for this simple example we only have one Kafka partition. If using this method it’s important to note that these results show the most recent offset ingested, and +1 must be added to specify where Spark should start reading.

SELECT partition, MAX(offset)
FROM neil_test_catalog.streaming.json_bronze
GROUP BY partition

Finding via Spark Declarative Pipelines Checkpoints

Another way to retrieve your offset information is to query SDP’s /checkpoints/ folder that your pipeline uses to track state and progress. For more detailed information on checkpoints and how Spark Structured Streaming achieves exactly-once processing, check out this blog: Inside Delta Lake’s Idempotency Magic: The Secret to Exactly-Once Spark.

First, find your streaming table’s storage location via DESCRIBE DETAIL.

Note: If your table is Unity Catalog-managed, this method requires direct read access to the table’s managed storage location.

Using the location, you can append “/_dlt_metadata/checkpoints/your_table_name/” to find the most recent streaming query context (the greatest number). For this streaming table the max result is 8 as shown below.

path = “s3://....”
metadata_path = path + “/_dlt_metadata/checkpoints/neil_test_catalog.streaming.json_bronze/”
display(dbutils.fs.ls(metadata_path))

Within the numbered subfolder under your table name, you will see /offsets/ and /commits/ folders, each also containing numbered folders 0/, 1/, 2/ and so on. These folders represent streaming batches in SDP.

• offsets/N is a write‑ahead log entry written before processing batch N. It stores the end offsets (high‑water mark) for that batch — i.e., “read up to here” for each topic/partition.

• commits/N is only written after batch N has finished successfully.

Because an offsets/N entry can exist even if its corresponding commits/N is missing (a batch started but never committed), you should follow these steps to determine where to retrieve your startingOffsets.

List the batch IDs in both folders:

offsets_path = path + "/_dlt_metadata/checkpoints/your_table_name/8/offsets/"
commits_path = path + "/_dlt_metadata/checkpoints/your_table_name/8/commits/"
display(dbutils.fs.ls(offsets_path))
display(dbutils.fs.ls(commits_path))

Let:

• commit_batches = all numeric batch IDs under commits/

• max_commit_batch = largest value in commit_batches

Use max_commit_batch as the last fully committed batch, and read the matching offsets file:

Always read offsets/max_commit_batch to get the correct startingOffsets JSON.

Ignore any higher batch ID that appears only under offsets/ but not commits/. That batch started but never finished, so if you treat its offsets as your starting position, Spark will behave as if that data has already been read and will skip it rather than processing it.

In my example, both the /offsets/ and /commits/ folders contain only batches 0 and 1, so using max_commit_batch, we read startingOffsets from /offsets/1.

metadata_path = path + “/_dlt_metadata/checkpoints/neil_test_catalog.streaming.json_bronze/8/offsets/1/”
display(dbutils.fs.head(metadata_path))

{”batchWatermarkMs”:0,”batchTimestampMs”:1771439553201,”conf”:{...}}
{”neil_struct_topic”:{”0”:4}}

Notice the final line is our starting partition:offset information in the exact format we created manually above. Spark tracks the next offset to read, so you do not need to increment +1 via this method. Again, if our topic had multiple partitions it might look like:

{”neil_struct_topic”:{”0”:4, “1”:6}}

Step 4: Update pipeline definition

Now it’s time to apply the actual logic changes that prompted this process. For my example, I add variant support to table properties:

table_properties={”quality”: “bronze”, “delta.feature.variantType-preview”: “supported”}

And cast the Struct columns to Variant:

return (
        parsed
            .withColumn(”event”, expr(”parse_json(to_json(event))”))
            .withColumn(”device”, expr(”parse_json(to_json(device))”))
        )

Step 5: Full Refresh Table

With pipeline logic updated, it’s time to run with Full refresh to wipe the target table and ingest the new Kafka records starting at our specified offsets. The resulting table will contain only the records we did not back up via Step 2.

To do this, we run our pipeline with full refresh after adding our startingOffsets (line 28). Here is the final pipeline definition.

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)

import dlt
from pyspark.sql.functions import current_timestamp, col, from_json, expr

SERVERS = “REDACTED”
TOPIC = “neil_struct_topic”

# Explicit JSON schema for predictable demos (no schemaLocationKey)
SCHEMA_DDL = “”“
  user STRING,
  email STRING,
  country STRING,
  device STRUCT<os: STRING, model: STRING, geo: STRUCT<lat: DOUBLE, lon: DOUBLE>>,
  event STRUCT<name: STRING, seq: BIGINT, ts: STRING>
“”“

@dlt.table(
    name=”json_bronze”,
    comment=”Raw Kafka payload with explicit JSON schema and rescued data”,
    table_properties={”quality”: “bronze”, “delta.feature.variantType-preview”: “supported”}
)
def json_bronze():
    df = (
        spark.readStream
            .format(”kafka”)
            .option(”kafka.bootstrap.servers”, SERVERS)
            .option(”kafka.security.protocol”, “SSL”)
            .option(”subscribe”, TOPIC)
            .option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)
            .load()
    )

    parsed = (
        df.selectExpr(”CAST(value AS STRING) AS json_str”, “topic”, “partition”, “offset”, “timestamp”)
          .select(
              from_json(
                  col(”json_str”),
                  SCHEMA_DDL,
                  options={”rescuedDataColumn”: “_rescued_data”}  # capture type mismatches/new fields
              ).alias(”data”),
              “topic”, “partition”, “offset”, “timestamp”
          )
          # data.* includes _rescued_data already; do not reselect it to avoid duplicate column error
          .selectExpr(”data.*”, “topic”, “partition”, “offset”, “timestamp AS kafka_timestamp”)
          .withColumn(”ingestion_ts”, current_timestamp())
    )

    return (
        parsed
            .withColumn(”event”, expr(”parse_json(to_json(event))”))
            .withColumn(”device”, expr(”parse_json(to_json(device))”))
        )

On the pipeline page:

Here’s the result: user-5 and user-6 as expected:

Step 6: Insert Historical Records

To complete our intended result table, insert historical data from the backup table, ensuring the data matches the new table format (cast columns, etc.):

INSERT INTO neil_test_catalog.streaming.json_bronze
SELECT user, email, country, to_variant_object(device), to_variant_object(event), _rescued_data, topic, partition, offset, kafka_timestamp, ingestion_ts
FROM neil_test_catalog.streaming.json_bronze_backup

Our final result. Your pipeline may be resumed.

user:string
email:string
country:string
device:variant
event:variant
_rescued_data:string
topic:string
partition:integer
offset:long
kafka_timestamp:timestamp
ingestion_ts:timestamp

Step 7: Revert startingOffsets

To avoid pipeline failures in the case of a future Full Refresh, revert startingOffsets to its prior value.

.option(”startingOffsets”, “earliest”)

Frequently Asked Questions

Q: Why can’t I just run a standard Full Refresh on the pipeline?

A: A Full Refresh clears the target table and re-reads the source from the beginning. If your Kafka topic has a retention policy (TTL), data that has "aged out" of the topic will be permanently lost because it no longer exists in the source to be re-read.

Q: Why is startingOffsets necessary if I have a backup?

A: While the backup saves your history, startingOffsets ensures your pipeline resumes reading exactly where the backup stopped. Without this explicit instruction, the pipeline might default to "earliest" (reading only what remains in Kafka, creating a gap) or "latest" (skipping data that arrived during the maintenance window).

Q: Is this process required for "Append-Only" tables?

A: Generally, yes, if you need to restructure the existing table. If you are only adding new columns that are nullable, you might rely on schema evolution, but fundamental type changes usually require the table to be rewritten.

• The API calls to the FMs were synchronous, so QPS would have been Medieval indeed

• The Lakebase instance was provisioned, so it would always be accruing costs even without usage

• There was no UI, so you or your admin would be spending hours fiddling with thousand line SQL queries for enterprise use cases

But no matter, we’ve had a Renaissance!

The repo got three meaningful updates and a handful of smaller ones that collectively move this from being technically functional to something a medium enterprise team might actually want to use on Databricks.

Subscribe now

Quick Refresher

The original solution uses a custom MLflow model serving endpoint as a proxy between your users and whatever foundation model they’re calling. Before the request hits the FM, the endpoint checks two Lakebase tables: one for the user’s token limit and another for how many tokens they’ve already burned through. If they’re over budget, the request ends up like John the Baptist in the cover image. If not, it goes through to the FM and the usage is written back to Lakebase along with the response and remaining balance.

Change 1: Autoscaling Lakebase

The original code used a Provisioned Lakebase instance because that was the only option at the time, but it’s going away and we have something better. Autoscaling Lakebase.

Swapping to Autoscaling Lakebase means you can set minimum and maximum scaling bands, and if you don’t need a high availability instance, it will also allow you to scale to zero during times of no use.

This is the smallest change architecturally, but it’s nice not to pay for compute we don’t need.

Change 2: ResponsesAgent + Async FM Calls

The original code used mlflow.pyfunc.PythonModel and called the FM endpoint via requests.post(), which is synchronous and blocking. Only one request can be handled at a time per unit of concurrency. Which meant the endpoint that was supposed to help you manage costs via budgeting is throttling your throughput instead. While I suppose that is one way to reduce token costs, it’s not very useful.

The new version replaces the PythonModel with a ResponsesAgent and swaps requests for httpx.AsyncClient inside an async def predict_stream(). Now multiple FM calls can be in flight simultaneously and the serving endpoint isn’t waiting on one user’s 20-second Claude response before it can look at the next request in the queue.

The core logic now lives in a standalone rate_limiter_agent.py decoupled from the notebook. The public API is much cleaner:

  agent = TokenRateLimiterAgent(
      db_config={...},
      workspace_client=WorkspaceClient(),
      endpoint_name="ep-your-endpoint",
      group_members={
          "data-science-team": ["andrea@company.com", "john@company.com"],
      },
  )

  # Before calling the FM:
  quota = agent.check_quota("andrea@company.com", "databricks-claude-sonnet-4-5")
  if not quota["allowed"]:
      # Return 429 or block the request
      ...

  # After the FM call completes:
  agent.log_usage(
      user_name="andrea@company.com",
      model_name="databricks-claude-sonnet-4-5",
      prompt_tokens=1200,
      completion_tokens=350,
      request_id="req-abc123",
  )

You can drop this into any existing pipeline without touching the notebook, which certainly helps if you’re integrating this into something that already has its own serving infrastructure.

Change 3: An Actual Frontend

The original had no management UI, which meant you had to set limits by writing manual SQL queries for any new change to your budgeting policy. Not very convenient.

The new repo ships a full Databricks App: a React + FastAPI application that deploys alongside your serving endpoint and gives administrators a no-code interface for setting granular budgets. How granular you ask? Any combination of:

• A user, service principal, or group

• Calling any FM, list of FMs, or across all FMs in the workspace

• That resets every X hours, days, weeks, months, or never

• Limited to a specified count of tokens or dollars

  • Pre-populates token costs from Databricks documentation, but manually editable in case this changes or you have some kind of secret discount I don’t know about

  • This is another nice quality of life feature since tokens are not all created equally; GPT OSS 20B tokens cost about 100x less than GPT 5.4 tokens

The drop-downs auto-populate users, SPs, and groups as well as the Databricks Foundation Models.

It also comes with a handy monitoring dashboard so you can see usage over time, your top consumers, and the most popular models.

The App authenticates to Lakebase via a native Postgres role with a static password stored in Databricks Secrets, so there’s no OAuth token refresh to manage.

An Honest Conclusion

Is this production-grade for an org running thousands of concurrent end users? Maybe not. You might consider mini-batching requests at scale, but there will still be some amount of cost tracking overhead, and this gets more difficult at scale.

Is this production-grade for most actual enterprise teams who want to stop their power users from accidentally burning through their monthly token budget in a week? Yes. I think this solution really shines when you have dozens to hundreds of daily active users who might get greedy on Opus requests without some budget enforcement.

But don’t take my word for it; check it out for yourself. The code lives here and setup instructions are in the README.

Cheers and happy coding.

As a Specialist Solutions Architect at Databricks, I often hear the same questions from customers who are migrating dashboards from legacy BI tools to Databricks AI/BI Dashboards:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

In the first blog post in this series, I focused on the first two questions and showed how to recreate:

• context filters using parameters in dataset SQL, and

• “Only Relevant Values” filters using field filters and query-based parameters.

This blog tackles the third question: “How do we replace filter actions from existing dashboards when we click on a bar/segment/point?”

In many BI tools, this behavior is configured as a filter action (a click on a mark filters other views). In Databricks AI/BI Dashboards, the equivalent interactivity is split into two built-in features:

• Cross-filtering, where clicking a mark in one chart filters other charts on the same page that use the same dataset.

• Drill-through, where right-clicking a mark opens a target page filtered to that selection.

Once you combine cross-filtering and drill-through with the context and cascading patterns, you can reproduce most real-world filter-action workflows.

As before, I’ll use the built-in samples.tpch dataset. I’ve also published the companion dashboard, so you can follow along and inspect the configurations yourself.

Recap: TPCH Sales dataset

To keep examples concrete, we’ll use the TPCH sample data that ships with Databricks in the samples.tpch schema. I’ll reuse the same base dataset, TPCH Sales, from the first blog post, which joins tables region, nation, customer, orders, and lineitem, and computes revenue:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

In the companion dashboard, this is the TPCH Sales dataset. It and its derivatives are used by multiple pages:

• Context filter

• Cascading filters with field filters

• Cascading filters with query-based parameters

• Cross-filtering (new in this post)

• Drill-through details (new in this post)

• and others

Filter actions vs. cross-filtering and drill-through

Before we build anything, it helps to align vocabulary.

In many traditional BI tools, filter actions are configured explicitly:

• You specify one or more source sheets.

• You specify one or more target sheets.

• You choose which fields are passed as filters.

• You choose what happens when the selection is cleared.

In Databricks AI/BI Dashboards, cross-filtering is implicit. You don’t turn it on in the visualization panel.

• It is automatically applied to supported visualization types that use the same dataset.

• When you click a bar, slice, or point, AI/BI adds a filter based on that value and re-runs all other visualizations on the page that share the dataset.

In Databricks AI/BI Dashboards, drill-through is also implicit, but slightly more structured:

• When you right-click a supported chart type, AI/BI shows Drill to → <target page> if there is another page in the dashboard that uses the same dataset.

• The target page opens with all visuals based on that dataset filtered to the selected segment, and any compatible filters on that dataset are auto-populated.

Conceptually:

• Cross-filtering ≈ a within-page filter action.

• Drill-through ≈ a navigation filter action (summary → details).

The rest of the post walks through how to configure your pages so these implicit behaviors “just work”.

1. Recreating within-page filter actions with cross-filtering

A very common dashboard pattern is:

• A summary bar chart (for example, total revenue by nation).

• One or more supporting charts (for example, revenue share by region, or a donut chart for mix).

• A filter action so clicking a bar filters the other visuals.

In AI/BI Dashboards, this becomes cross-filtering on top of the TPCH Sales dataset.

1.1. When cross-filtering is applied

Cross-filtering is applied automatically when all of the following are true:

• The visualizations are on the same page.

• The visualizations use the same dataset (for example, TPCH Sales).

• The visualization type is one of the supported chart types: Bar, Box plot, Heatmap, Histogram, Pie, Scatter, or Point map.

If those conditions are met, there is nothing to enable. Clicks on supported charts become filters for any other visualizations on the page that use that dataset.

1.2. Build the Cross-filtering page

In your AI/BI dashboard, add a page named Cross-filtering.

On this page:

1. Add a bar chart: Revenue by nation

   • Visualization: Bar

   • Dataset = TPCH Sales

   • X axis: nation

   • Y axis: SUM(revenue)

2. Add a pie: Revenue by region

   • Visualization: Pie

   • Dataset = TPCH Sales

   • Slice by (Color): region

   • Value (Angle): SUM(revenue)

3. Add filters (optional)

   • Add a Region filter on TPCH Sales.region

   • Add a Nation filter on TPCH Sales.nation

These are just standard filter widgets. There is no “cross-filtering” toggle anywhere in the configuration.

In the companion dashboard, this page is already built for you.

1.3. Use cross-filtering on the charts

Try the following workflow:

1. Select two regions, AFRICA and ASIA, on the Revenue by region chart.

2. Click the bar for JAPAN in the Revenue by nation chart.

Because cross-filtering is automatic for supported charts that share a dataset:

• AI/BI adds filters Nation: JAPAN and Region: AFRICA, ASIA to the TPCH Sales dataset for this page.

• Both charts get updated accordingly.

You can Reset all to default and try a different filter combination.

From a migration perspective, this answers a common question: “Can clicking a bar automatically update the rest of the dashboard in AI/BI?” Yes – when charts share a dataset and use supported visualization types, cross-filtering works implicitly with no additional configuration.

2. Recreating across-page filter actions with drill-through

Another classic summary → detail filter-action pattern is:

• A summary view (for example, revenue by nation).

• A detail view (for example, individual orders).

• A filter action that passes the selected value into the detail sheet as a filter.

In AI/BI Dashboards, this is implemented as drill-through.

2.1. How drill-through is applied

Drill-through shows up as a right-click option when several conditions are satisfied.

• The source chart is a supported type: Bar, Box plot, Heatmap, Histogram, Pie, Scatter, or Point map.

• There is at least one target page in the same dashboard where:

  • At least one visualization uses the same dataset as the source chart.

  • The field you click on has a compatible filter or column on the target page.

In recent AI/BI releases, drill-through no longer requires an explicit target filter; any visualization based on the same dataset as the source selection is filtered automatically, and filters (if they exist) are populated with the drilled values.

There is no drill-through toggle in the widget side panel. Once the above conditions are true, AI/BI surfaces Drill to → <page name> in the context menu automatically.

2.2. Build the Drill-through details page

In the same dashboard, add another page named Drill-through details.

On this page:

1. Add a detail table visualization

   • Visualization: Table

   • Dataset = TPCH Sales

   • Columns: region, nation, customer_id, customer_name, order_id, order_date, revenue

2. Add filters (optional)

   • Add a Region filter on TPCH Sales.region

   • Add a Nation filter on TPCH Sales.nation

   • Add a Customer filter on TPCH Sales.customer_id

Again, there is no special drill-through configuration here – just a normal page that uses the same dataset.

In the companion dashboard, this page is already built for you.

2.3. Drill from summary to details

Go back to the Cross-filtering page and right-click a bar in the Revenue by nation chart (for example, UNITED STATES).

After clicking Drill to → Drill-through details, AI/BI opens the Drill-through details page and filters all visualizations based on TPCH Sales to Nation: UNITED STATES.

From the user’s perspective, this feels almost identical to a filter action in traditional dashboards that navigates from a summary sheet to a detailed sheet with the selected country carried over.

Cross-filtering vs. drill-through vs. filter widgets

Across this blog and the first blog, you now have three main interaction tools in AI/BI Dashboards:

1. Filter widgets

   • Context filters, cascading filters, and query-based parameters.

   • Best for primary, always-visible controls like Region, Date, Product.

2. Cross-filtering

   • Click data points in a supported visualization to filter other charts on the page that use the same dataset.

   • Best for ad-hoc exploration, answering questions like “What happens if I focus only on this region?” and “Which nations are driving that spike?”.

3. Drill-through

   • Right-click a mark to open another page already filtered to that selection.

   • Best for guided summary-to-detail flows where you don’t want to cram everything onto one page.

A simple migration rule of thumb:

• Use filter widgets to rebuild the core filter panels from your existing dashboards.

• Use cross-filtering to automatically filter other visualizations on the page based on chart interactions.

• Use drill-through to replace “go to sheet” filter actions and connect high-level KPIs to detail pages.

Summary and what’s next

In this second blog post in the series, we answered: “Do you support filter actions when I click on a bar or a point?”

The short answer is yes:

• Cross-filtering lets viewers click on supported charts to filter all other visualizations on the same page that share a dataset – no configuration required.

• Drill-through lets viewers right-click a mark and open another page where visuals on the same dataset are already filtered to the selected values, and any matching filters are pre-populated.

Combined with the patterns from the first blog post – context filters and cascading “Only Relevant Values” filters – you now have a robust toolkit for recreating the interactive filtering experience your users expect from traditional dashboards inside Databricks AI/BI Dashboards.

The companion dashboard now includes: a context filter page, a field-based cascading page, a query-based cascading page, a cross-filtering page, and a drill-through details page. You can import it into your workspace and adapt the patterns to your own datasets.

In the next post in this series, I’ll tackle the remaining migration question: “How do we do user-based filtering and row-level security in AI/BI Dashboards?”

The first section explains how replication from Postgres works, and how to configure it. The second part show how to configure Lakeflow pipelines in your Databricks workspace to ingest from Postgres.

How PostgreSQL logical replication works

PostgreSQL writes every change (insert, update, delete) to the Write-Ahead Log (WAL) before applying it. Logical replication decodes that WAL stream into a human-readable format that consumers can subscribe to.

Three objects are involved:

1. WAL level

Logical replication requires wal_level = logical. On AWS RDS, set rds.logical_replication = 1 in your parameter group (requires a reboot).

2. Publication

A publication is a named filter over which tables to expose for replication. It lives inside a single database.

-- Replicate specific tables
CREATE PUBLICATION mfg_paloalto_cdc_pub FOR TABLE iot1.devices, iot1.alerts;

-- Or replicate all tables in the database
CREATE PUBLICATION mfg_paloalto_cdc_pub FOR ALL TABLES;

One publication per database is typical. If you have multiple PostgreSQL databases on the same server, each needs its own publication.

3. Replication slot

A replication slot is a cursor into the WAL. PostgreSQL retains WAL segments until the slot consumer confirms it has processed them (confirmed_flush_lsn). This guarantees the consumer never misses a change, even if it disconnects temporarily.

SELECT pg_create_logical_replication_slot(’databricks_mfg_paloalto_slot’, ‘pgoutput’);

Key facts about slots:

• Slots are server-wide — the name must be unique across the entire PostgreSQL instance, not just within a database.

• One slot per consumer — a slot can only be consumed by one connection at a time. If a second pipeline tries to use the same slot, it will be rejected.

• Unacknowledged WAL accumulates — if a pipeline is stopped and not consuming, the slot holds back WAL. Monitor pg_replication_slots.lag to avoid disk pressure.

Replication user

Create a dedicated user with replication privileges and read access to the tables being replicated:

CREATE USER databricks_replication WITH PASSWORD ‘...’ REPLICATION;

-- On RDS, also grant the rds_replication role
GRANT rds_replication TO databricks_replication;

GRANT CONNECT ON DATABASE mydb TO databricks_replication;
GRANT USAGE ON SCHEMA iot1 TO databricks_replication;
GRANT SELECT ON ALL TABLES IN SCHEMA iot1 TO databricks_replication;

Replica identity

For CDC updates and deletes to include the old row values (needed to identify which row changed), each table needs a replica identity:

-- Default: uses the primary key (recommended when a PK exists)
ALTER TABLE iot1.devices REPLICA IDENTITY DEFAULT;

-- Full: includes all columns (required when there is no PK)
ALTER TABLE iot1.devices REPLICA IDENTITY FULL;

One slot per consumer, not per catalog

The Databricks docs suggest creating one replication slot per source catalog. A more precise rule is: one replication slot per consumer (ingestion pipeline).

If you want to replicate the same database to two different workspaces, or run multiple pipelines in a test environment, each needs its own slot. Using the same slot across multiple pipelines will cause one of them to lose data.

Multiple databases on the same server

PostgreSQL publications are per-database, but replication slots are server-wide. If you have two databases (mfg_paloalto and mfg_austin) on the same server, create two replication slots with different names.

Server
├── Database: mfg_paloalto
│   └── Publication: mfg_paloalto_cdc_pub        (per-database)
├── Database: mfg_austin
│   └── Publication: mfg_austin_cdc_pub           (per-database)
│
├── Slot: databricks_mfg_paloalto_slot             (server-wide, for mfg_paloalto DB)
└── Slot: databricks_mfg_austin_slot               (server-wide, for mfg_austin DB)

Databricks side

Three entities need to be created in order. Below shows the json sent using the Databricks CLI.

1. Unity Catalog connection

The connection is at the server level — no database name. One connection can serve all databases on the same PostgreSQL instance.

// connection.json
{
  “name”: “my-postgres-connection”,
  “connection_type”: “POSTGRESQL”,
  “options”: {
    “host”: “myinstance.abc123.us-east-1.rds.amazonaws.com”,
    “port”: “5432”,
    “user”: “databricks_replication”,
    “password”: “<password>”
  }
}

databricks connections create --json connection.json

A single connection can be used to ingest from multiple databases, as long as the user has permissions.

2. Gateway pipeline

The gateway pipeline runs continuously on a classic cluster. It connects to PostgreSQL, reads the WAL via the replication slot, and buffers raw CDC events into a storage schema in Unity Catalog.

// gateway.json
{
  “name”: “my-postgres-gateway”,
  “catalog”: “my_catalog”,
  “schema”: “cdc_gateway_storage”,
  “channel”: “CURRENT”,
  “continuous”: true,
  “gateway_definition”: {
    “connection_name”: “my-postgres-connection”,
    “gateway_storage_catalog”: “my_catalog”,
    “gateway_storage_schema”: “cdc_gateway_storage”
  }
  “clusters”: [
    {
      “label”: “default”,
      “driver_node_type_id”: “r5.xlarge”,
      “node_type_id”: “r5.xlarge”,
      “autoscale”: {
        “min_workers”: 1,
        “max_workers”: 5
      }        
    }
  ]
}

Setting min_workers = max_workers = 0 will give you a driver-only cluster, if your CDC stream is low volume.

databricks pipelines create --json gateway.json

Note the pipeline_id returned — it is needed for the ingestion pipeline.

3. Ingestion pipeline

The ingestion pipeline reads buffered events from the gateway and writes them as Delta tables. It runs on serverless and is triggered (not continuous).

source_configurations maps each source database to its replication slot and publication. (Unfortunately the postgres source database is referred to as catalog.source_catalog which is a little confusing.) objects controls which schemas or tables to replicate and where they land.

The json below creates a single pipeline from two source schemas in different databases (mfg_paloalto and mfg_austin). Notice how the slot names correspond to the right database.

// ingestion.json
{
  “name”: “my-postgres-ingestion”,
  “catalog”: “my_catalog”,
  “schema”: “mfg_ingest_metadata”,
  “ingestion_definition”: {
    “ingestion_gateway_id”: “<gateway-pipeline-id>”,
    “source_type”: “POSTGRESQL”,
    “connection_name”: null,
    “objects”: [
      {
        “schema”: {
          “source_catalog”: “mfg_paloalto”,
          “source_schema”: “iot1”,
          “destination_catalog”: “my_catalog”,
          “destination_schema”: “mfg_paloalto_iot1”
        }
      },
      {
        “schema”: {
          “source_catalog”: “mfg_austin”,
          “source_schema”: “iot1”,
          “destination_catalog”: “my_catalog”,
          “destination_schema”: “mfg_austin_iot1”
        }
      }
    ],
    “source_configurations”: [
      {
        “catalog”: {
          “source_catalog”: “mfg_paloalto”,
          “postgres”: {
            “slot_config”: {
              “slot_name”: “databricks_mfg_paloalto_slot”,
              “publication_name”: “mfg_paloalto_cdc_pub”
            }
          }
        }
      },
      {
        “catalog”: {
          “source_catalog”: “mfg_austin”,
          “postgres”: {
            “slot_config”: {
              “slot_name”: “databricks_mfg_austin_slot”,
              “publication_name”: “mfg_austin_cdc_pub”
            }
          }
        }
      }
    ]
  }
}

databricks pipelines create --json ingestion.json

The top level catalog.schema (my_catalog.mfg_ingest_metadata above) hold event logs and checkpoints for all the pipelines.

The above create command also returns a pipeline id ingestion-pipeline-id) which can be used to trigger the pipeline.

Triggering a run

After creation, trigger an initial snapshot + CDC run:

databricks pipelines start-update <ingestion-pipeline-id>

The gateway pipeline should run continually. The ingestion pipeline can be run at whatever interval you want, including continually.

Summary checklist

Summary checklist

Step                    Scope                        Notes
────────────────────────────────────────────────────────────────────────────
wal_level = logical     Server                       Requires reboot on RDS
Replication user        Server                       One user can serve all databases
REPLICA IDENTITY        Per table                    Set before creating the publication
Publication             Per database                 One per database is typical
Replication slot        Per database, server-wide    One slot per ingestion pipeline
                        name
Network access          Server                       Allow Databricks cluster egress IP on port 5432

Today’s data teams face a common challenge: streaming data from applications to their Lakehouse requires maintaining complex infrastructure. The typical setup involves managing a message bus like Kafka, configuring connectors, monitoring pipelines, and dealing with significant operational overhead and costs—all just to move data from point A to point B.

Managed Bus Don’t Solve Everything

While Amazon Managed Streaming for Apache Kafka(Amazon MSK) removes the burden of managing servers, it doesn’t eliminate your responsibility for the message bus itself. You’re still on the hook for capacity planning, topic and partition design, producer and consumer tuning, monitoring and alerting, and upgrade timing. Managed services make these tasks less manual, but upgrades remain risky and the operational complexity persists.

Cost management is another pain point. Amazon Managed Streaming for Apache Kafka(Amazon MSK) bills can balloon quickly due to over-provisioned brokers, excess partitions, high replication factors, and long retention periods. AWS manages the infrastructure, but not your spending discipline—that’s still your problem.

What You Actually Need

A fully abstracted streaming service that eliminates cluster management entirely, letting you focus on building data products instead of babysitting message bus infrastructure.

The Challenge

A rapidly growing organization was processing massive device data volumes from Go applications. After essential first-level processing, they needed to stream data to their data lake for near real-time analytics. To avoid the complexity of managing Kafka or similar message bus infrastructure, they took a shortcut: direct writes to their data warehouse with append-only inserts.

Initially simple, this approach quickly hit walls. As volumes grew, they vertically scaled, then horizontally distributed producers across multiple warehouse instances. Small but relentless queries created network bottlenecks. Excessive delta commits from numerous producers killed throughput. They hit soft limits on connections and write operations. To keep data flowing, they over-provisioned compute—watching costs balloon without proportional gains.

The Solution

Zerobus Ingest provided the purpose-built ingestion layer they needed. Their Go applications integrated the SDK with minimal code changes—same append-only pattern, properly architected. The Write-Ahead Logging (WAL) based system handled buffering and batching, while automatic recovery managed network issues that previously caused data loss.

Data now lands directly in Delta tables, eliminating the warehouse intermediary. The result: lower latency, higher throughput, dramatically reduced costs, and one less system to manage. They got the simplicity of direct writes with the scalability of proper streaming—at a fraction of Kafka’s cost.

Overview

Zerobus Ingest is a fully managed, zero-configuration service that enables record-by-record data ingestion directly into Delta tables. No more intermediate message buses. No more complex configurations. Just point your application at an endpoint and start sending data. The Zerobus Ingest API buffers transmitted data before adding it to a Delta table. This buffering creates an efficient and durable ingestion mechanism that supports a high volume of clients with variable throughput.

Before:

After:

Features

Zerobus Ingest leverages a Write Ahead Log (WAL) architecture that enables it to store and acknowledge accepted records quickly, delivering low write latency for your applications. The system is backed by persistent disk storage where both the Write Ahead Log (WAL) and checkpoints are maintained, enabling several powerful capabilities:

Automatic Recovery - Network issues are handled transparently by the SDK. It automatically reconnects on transient failures and resends unacknowledged records without requiring any application-level error handling code.

Efficient Resource Management - Once data syncs successfully to Delta tables, Zerobus Ingest automatically cleans up Write Ahead Log (WAL) logs and metadata, freeing disk space for new data without manual intervention.

Schema Management - Automatic validation against your Delta table schema catches data quality issues at ingestion time, preventing malformed data from entering your Lakehouse.

SDKs

Users will interact with Zerobus Ingest through a dedicated SDK for their language of choice. The documentation and samples are out for Python SDK, Rust SDK and Java SDK . Both the Go and TypeScript SDKs for Zerobus Ingest are now publicly available. GRPC is the main communication mechanism for Zerobus Ingest.

Databricks documentation contains a well documented guide with sample clients in multiple languages. It guides you right from installing the SDK in your preferred language to creating a Protobuf definition and a sample usage.

Supported Formats

• Protocols: gRPC (primary), HTTP REST, Kafka wire format (coming soon)

• Data Formats: Protocol Buffers, JSON

TIPs

• Visit the table history on UC to get a sense of how frequently the table is updated

• Handle the two exceptions gracefully NonRetriableException, ZerobusException.

• Even though Zerobus Ingest periodically issues data file compactions, so you don’t need to worry about the small files

• Don’t forget to create a table with appropriate data types before you run the client

Components

1. Zerobus Ingest Server - Think of them as scalable stateful pod on K8s attached with an SSD disk(high IOPS). Its responsibilities include:

   • Schema validation of the message to the table.

   • Materializing the data in a timely manner to the target table.

   • Sending an acknowledgement to the client that the data is durable.

2. Smart Networking and orchestration - API proxy which distributes the streams to Zerobus Ingest servers per the target delta table and scales pods as the utilization nears the roof

3. Delta kernel - Record batch writer kicks off every 1-5 seconds, uses Delta kernel(uses Arrow) and writes the record batch to the delta table. Kicks in the PO compaction to avoid small files. Rust APIs hides all the complex details of the Delta protocol specification. Binding available for python.

4. Write-Ahead Log(WAL) - Records are immediately persisted to durable storage(think SSD disks with high IOPS) provided by the cloud platform your databricks is running on and is acknowledged in under 50ms. This guarantees durability even if something fails

Zerobus Ingest fits direct lakehouse writes with durable acknowledgments (no bus-style retention/multi-consumer). Zerobus Ingest it not a replacement of message bus in all scenarios. If you need message bus durability/retention or multiple subscribers, Event Hubs/Kafka is likely a safer choice.

Availability

Databricks only support single availability zone (single AZ) durability. This means Zerobus Ingest service may experience downtime. This might change soon.

Kafka still wins when

Despite the cost advantages of Zerobus Ingest Ingest, Kafka remains a better choice in following scenarios:

Exactly-once semantics requirements - For financial transactions, order processing, or other workflows where duplicate processing could cause serious issues, Kafka’s exactly-once delivery guarantees are critical. While Zerobus Ingest roadmap includes this feature, organizations that need it today must still rely on Kafka.

Ultra-low latency fan-out - If your use case requires multiple consumers reading the same stream with different processing logic, Kafka’s pub-sub model excels. Zerobus Ingest currently lacks the subscriber/consumer model that makes Kafka so powerful for fan-out patterns where one stream feeds multiple downstream applications.

Other Limitations

As of writing,

• Zerobus Ingest provides at-least-once delivery semantics, meaning each message will be delivered one or more times. It does not yet support exactly-once semantics. However, the duplicates can be handled using other Databricks and delta features.

• Zerobus Ingest currently supports writing only to managed Delta tables

• Schema evolution on target tables is not yet supported in Zerobus Ingest, so the table schema must match the incoming message structure.

• Each individual message is limited to a maximum size of 10 MB when processed through Zerobus Ingest.

What’s Next

Databricks is actively enhancing Zerobus Ingest with several key features in development. The roadmap includes exactly-once delivery semantics for stronger consistency guarantees, MQTT protocol support to broaden IoT and device connectivity options, comprehensive CDC pipeline capabilities that will handle updates and deletes in addition to inserts and subscriber/consumer model to enable more flexible data consumption patterns.

Enjoy streaming in a cost efficient and simplified manner!

Zerobus Ingest offers a compelling alternative to message bus in a lot of scenarios. While Kafka remains essential for complex streaming architectures, Zerobus Ingest closes the gap for straightforward ingestion use cases—delivering the reliability you need at a fraction of the cost and complexity.

The cost savings extend beyond infrastructure. Kafka expertise commands premium salaries, and maintaining distributed message bus systems requires dedicated engineering time that could be spent on higher-value work. Zerobus Ingest’s simplicity means junior engineers can manage what previously required highly skilled distributed systems expertise. When you factor in reduced operational overhead, lower training costs, and faster time-to-production, the economics become even more compelling. Sometimes the best architecture isn’t the most sophisticated—it’s the one that solves your problem.

FAQ

What is Zerobus Ingest?

A fully managed ingestion service for sending application data directly into Delta tables without operating a separate message bus.

When is Zerobus Ingest a good fit?

It works best for high-volume, append-only ingestion where the primary goal is to land data in the Lakehouse quickly and simply.

Does Zerobus Ingest replace a message bus for every use case?

No. If you need long retention, multiple downstream subscribers, or mature exactly-once semantics today, a message bus is still the better fit.

What delivery semantics does Zerobus Ingest provide today?

It currently provides at-least-once delivery, so downstream designs should account for possible duplicates.

What tables and formats are supported?

The draft states that Zerobus Ingest currently writes only to managed Delta tables and supports Protocol Buffers and JSON.

What are the main implementation requirements?

You need to create the target table in advance, align the incoming schema with the table schema, and handle SDK exceptions appropri

Reference

Stream 10+ GB/sec to your lakehouse in under 5 seconds with zero infrastructure overhead

Deep dive on Zerobus Ingest, now GA

• The Problem: AI agents generate massive volumes of trace data, but traditional observability tools make that data expensive to retain, difficult to govern, and hard to use in evaluation and analytics workflows.

• The Solution: MLflow now supports writing OpenTelemetry (OTEL) traces directly to Unity Catalog tables via a fully managed, serverless ingestion path.

• The Benefit: By landing traces directly in the Lakehouse, teams get governed, analytics-ready observability data with long-term retention, unified evaluation and monitoring workflows, and no OTEL infrastructure to operate.

• The Outcome: Production traces become immediately usable for analysis and evaluation, enabling faster iteration loops between real-world usage, model evaluation, and continuous improvement.

Why AI Tracing Breaks Traditional Observability

As AI applications move into production, traces become one of the clearest ways to understand how agents actually behave by capturing prompts, tool calls, responses, latency, and execution paths. Without strong tracing, it’s hard to understand why agents behave the way they do, making debugging, evaluation, and governance much more difficult.

The challenge isn’t that observability platforms can’t ingest this data. It’s that AI traces quickly become valuable beyond debugging. Teams want to retain them longer, analyze them with SQL, join them with business and model data, and reuse them for evaluation and monitoring. When traces live only inside observability systems, that flexibility is limited, governance becomes fragmented, and moving data into analytics workflows often requires extra pipelines and duplication, especially when sensitive prompt data is involved.

MLflow and OTEL Trace Ingestion

Databricks now supports writing MLflow traces directly to Unity Catalog using the OpenTelemetry (OTEL) format. In practice, this means traces can be ingested in real time and stored in Delta tables, where they benefit from the same scalability, governance, and tooling as the rest of your data.

This changes how teams can use trace data:

• Real-time ingestion with practical retention: Traces can be written as they’re generated at high throughput (GBs/sec) and retained long-term without the cost pressure typically associated with observability platforms.

• Analyze and govern using the Lakehouse: Once traces are tables, you can treat them like any other dataset: query them with SQL, build dashboards, run ETL pipelines, use tools like Genie, and apply governance controls such as PII masking.

• Use the full MLflow evaluation stack: Persisting traces in Unity Catalog removes typical experiment constraints (such as trace caps), making it easier to run large offline evaluations, monitor production systems, and continuously improve quality as workloads grow.

The Engineering trade-off: SaaS vs. Lakehouse

So why not rely entirely on a SaaS observability tool?

1. Retention economics: Agents generate massive text payloads. Storing this data in Delta Lake on object storage is often significantly more cost-effective than SaaS-based retention models.

2. The PII deadlock: Sending raw prompts to third-party platforms can create InfoSec friction. Keeping traces inside Unity Catalog helps maintain data sovereignty and simplifies governance.

3. Analytics, not just telemetry: SaaS tools are strong for operational metrics like latency, but the Lakehouse gives you something different: an analytics and AI engine. You can join traces with business data — revenue, conversions, customer outcomes — to understand real impact, not just system health. Furthermore, the Lakehouse enables you to apply AI directly to your traces, allowing for advanced use cases like classifying user interactions as ‘good’ or ‘bad,’ and building evaluation frameworks to continuously improve system quality.

Architecture: Serverless OpenTelemetry Ingestion

MLflow tracing can use the OpenTelemetry (OTEL) standard, which separates instrumentation from storage. In a typical OTEL deployment, teams are responsible for running collector fleets, scaling agents, handling backpressure, and managing reliability.

Databricks removes that operational layer by providing a managed OpenTelemetry endpoint, transparently powered by Zerobus. Zerobus is a serverless ingestion engine that enables applications to stream data directly into Delta tables using a gRPC API. Applications can easily export spans, logs, and metrics from any OTEL-compatible client directly to Unity Catalog tables, where the data is stored in Delta format. Zerobus acts as the telemetry pipeline, handling ingestion and durability so teams don’t have to operate their own collectors.

From there, traces become first-class data in the Lakehouse, powering MLflow evaluations and monitoring, ad-hoc SQL analysis, dashboards, and downstream analytics. This creates a continuous improvement flywheel where production behavior feeds evaluation and analysis, which in turn drives faster iteration and better agent performance.

Tutorial: Wiring Traces into the Lakehouse

Sample agent: Support manager assistant

For this blog, we’ll create a simple support manager assistant that we can use to demonstrate tracing end-to-end. The agent can be deployed outside of Databricks, as we’ve done here, highlighting that trace ingestion is decoupled from where the agent runs.

We built a LangGraph agent powered by a Databricks-hosted Claude Sonnet 4 model for reasoning and response generation. The agent calls a Genie Space as a tool, which you can deploy here.

When a user asks a data-driven question, the agent invokes Genie through the MCP tool API. Genie translates the request into SQL, executes it against the support dataset, and returns the result. The agent then summarizes the findings and provides actionable takeaways for a support manager.

Setting up MLflow tracing with UC

Before instrumenting the agent, we first configure MLflow to store traces in Unity Catalog. This involves creating the underlying OpenTelemetry tables and linking them to an MLflow experiment so traces can be searched, analyzed, and annotated from the UI. Start by identifying (or creating) a SQL warehouse and an MLflow experiment, then use the MLflow Python library to create the Unity Catalog tables and link the schema to the experiment. For full steps, follow the docs here.

import os
import mlflow
from mlflow.entities import UCSchemaLocation
from mlflow.tracing.enablement import set_experiment_trace_location

mlflow.set_tracking_uri("databricks")

os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "<warehouse-id>"

experiment_name = "<experiment-name>"
catalog_name = "<catalog>"
schema_name = "<schema>"

experiment_id = mlflow.create_experiment(name=experiment_name)

set_experiment_trace_location(
    location=UCSchemaLocation(
        catalog_name=catalog_name,
        schema_name=schema_name,
    ),
    experiment_id=experiment_id,
)

This setup creates Unity Catalog tables for spans, logs, and metrics. Once traces begin flowing, the MLflow service also creates Databricks views that transform the underlying OpenTelemetry data into an MLflow-friendly format for easier querying and analysis. These include:

• mlflow_experiment_trace_otel_spans: detailed execution steps for each request

• mlflow_experiment_trace_otel_logs: structured events such as metadata, tags, and assessments

• mlflow_experiment_trace_otel_metrics: numerical telemetry captured during execution

• mlflow_experiment_trace_metadata: MLflow tags, metadata, and assessments grouped by trace ID

• mlflow_experiment_trace_unified: a consolidated view that assembles all trace data into a single record per trace. For better performance at scale, consider converting it to a materialized view with incremental refresh.

After configuring the trace destination, agent instrumentation remains the same. You can do automatic and/or manual tracing as described here. In our example, we rely on mlflow.langchain.autolog() to capture the detailed LangGraph execution (model calls and tool calls). We also wrap the entrypoint with @mlflow.trace to establish a request-level root span, allowing each invocation to be observed as a single end-to-end execution.

Inspecting a sample trace

Now that the agent is instrumented and traces are flowing into Unity Catalog, let’s look at a real execution.

For this example, we asked the Support Manager Assistant:

“Which support engineer should I put up for promotion?”

The agent evaluated the request, called the Genie space multiple times to gather supporting data, and returned a recommendation based on performance metrics.

While the response looks straightforward, the trace reveals the underlying execution path that produced it. In the MLflow experiment, we can see each of the tool calls as well as the reasoning logic of our claude sonnet model. We can see that it called the genie space tool three times before putting together a final answer.

We can click through each of the individual steps to study the inputs and outputs.

Because traces are stored as Delta tables, they can be queried like any other dataset. We can start with the mlflow_experiment_trace_unified view, where we will find a record that includes the request, response, trace metadata, and an array of the spans.

Beyond Debugging: Analytics on Trace Data

Now that traces are stored in Unity Catalog, they become immediately available for both batch and streaming analytics.

Governance in Unity Catalog

Prompts and responses, however, often contain sensitive information, so treating trace data as governed data is critical. By storing it in Unity Catalog, traces inherit fine-grained access controls, from catalog and schema permissions to column masking and row-level filtering, enabling secure, production-ready analytics without limiting flexibility.

Once access is established, teams can securely run ad-hoc analytics by querying the underlying tables and views with SQL, as we did above. We can also build ETL pipelines, in addition to dashboards and genie spaces, for actionable business insights.

Dashboards

One of the most powerful aspects of having traces in Unity Catalog is that we aren’t locked into a vendor’s rigid, pre-canned views. Because the traces are in Delta tables, we can build custom dashboards that reflect our specific business logic, not just generic system health.

Using AI/BI Dashboards, we built an AI Operations Center that sits directly on top of our trace tables. This dashboard provides a unified view of our application performance, costs, and reliability. Instead of learning a proprietary query language, we just wrote standard SQL (with the help of AI) to extract exactly what we needed.

Here are some key capabilities this unlocked:

Custom Cost & Token Analysis
Generic “cost” metrics are rarely accurate because every team negotiates different rates or uses fine-tuned models with unique pricing. Since we control the SQL, we embedded our specific pricing logic directly into the query. Our dashboard tracks token usage by model type (e.g., GPT-4o vs. Claude 4 Sonnet) and applies our contract-specific rates to calculate a precise Estimated Cost per Trace. This lets us spot expensive outliers immediately—like a single complex query that costs $0.50 due to a retrieval loop.

Component-Level Performance

High-level latency metrics often hide the real culprit. Is the bottleneck the LLM or is it the Genie space retrieval? We built a “Tool Performance” widget that breaks down latency (P50, P99) and error rates for every individual tool in our agent (e.g., retrieve_docs vs. generate_response). This allows us to pinpoint exactly which step in the chain is degrading the user experience.

Genie spaces

Both business and technical stakeholders often want to explore agent behavior without writing SQL. By exposing trace tables through Genie, teams can enable natural-language analysis over their telemetry data, allowing users to ask questions about performance, tool usage, latency, and model behavior directly. In our example, this could include questions such as:

• What types of requests require escalation?

• Are tool retries increasing?

• Which queries trigger the most complex execution paths?

ETL pipelines

Because traces are stored as Delta tables, they can feed downstream ETL pipelines just like any other dataset. By enabling Change Data Feed (CDF), teams can process trace data incrementally, either in batch or streaming, without repeatedly scanning entire tables.

This makes it possible to operationalize observability. For example, a pipeline could monitor trace patterns and trigger alerts when latency exceeds defined thresholds, tool failures spike, or token usage deviates from expected baselines. These signals can then feed dashboards, notification systems, or automated remediation workflows.

Importantly, this complements real-time protections such as AI Guardrails. While guardrails enforce policy at request time, ETL pipelines create a feedback loop, helping teams analyze trends, refine policies, and continuously improve agent performance.

Closing the Loop: From Production Traces to Evaluation

Once traces are available, they can power the full MLflow 3 evaluation stack, enabling teams to measure, improve, and maintain the quality of their AI applications across the entire lifecycle. Evaluation and monitoring build directly on tracing, allowing the same telemetry captured during development, testing, and production to be scored using LLM judges and custom metrics.

Evaluate during development using AI Judges

MLflow allows us to run evaluations against an evaluation dataset, applying built-in or custom judges to score response quality. One effective approach is to bootstrap this dataset from real traces. Because these prompts originate from actual user interactions, they better represent the scenarios your agent must handle compared to synthetic test cases.

Below, we create an evaluation dataset from recently captured traces. MLflow uses a SQL warehouse to search and materialize dataset records, so be sure to configure the warehouse ID in your environment.

import os
import mlflow
import mlflow.genai.datasets
import time

# Required for dataset operations
os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = MLFLOW_TRACING_SQL_WAREHOUSE_ID

DATASET_NAME = f"{CATALOG_NAME}.{SCHEMA_NAME}.support_management_chatbot_traces"

# Create (or load) the dataset
try:
    eval_dataset = mlflow.genai.datasets.create_dataset(name=DATASET_NAME)
except Exception:
    eval_dataset = mlflow.genai.get_dataset(name=DATASET_NAME)

# Pull recent traces (example - from yesterday)
yesterday = int((time.time() - 60 * 60 * 24) * 1000)

traces_df = mlflow.search_traces(
    filter_string=f"attributes.timestamp_ms > {yesterday}",
    order_by=["attributes.timestamp_ms DESC"],
)

# Merge traces into the dataset
eval_dataset = eval_dataset.merge_records(traces_df[["inputs"]])

With the dataset in place, we can define the judges that will score our application. MLflow provides a set of built-in judges, and also allows us to define custom guidelines tailored to our agent’s expected behavior.

from mlflow.genai.scorers import RelevanceToQuery, Safety, Guidelines

# Define judges
agent_judges = [
    RelevanceToQuery(),
    Guidelines(
        name="analytical_correctness",
        guidelines="The response must correctly interpret the data and avoid unsupported conclusions.",
    ),
    Guidelines(
        name="actionable_support_insights",
        guidelines="The response must provide at least one concrete, data-backed recommendation.",
    ),
    Guidelines(
        name="performance_management",
        guidelines="The response should not recommend admonishing or firing employees.",
    ),
    Safety(),
]

# Run evaluation
eval_results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=predict_fn,
    scorers=agent_judges,
)

eval_results

And we can now see the results in the MLflow experiment.

Production monitoring

Development evaluations help us validate behavior before release, but production monitoring shows us how the application performs with real users. MLflow can automatically evaluate live traces using the same judges, helping us quickly detect regressions, drift, and emerging failure patterns. This turns evaluation from a one-time task into an ongoing practice as the application evolves.

Frequently Asked Questions (FAQ)

• Can I use this for agents running outside of Databricks?

  Yes, the agent can be running anywhere. In fact the support assistant agent example that was used for this blog is deployed locally.

• What are the throughput and storage limits of this solution?

  The ingestion throughput limit is 200 QPS today. There is no limit on storage. Previous limits on traces per experiment are no longer applicable. If you need higher throughput limits, please reach out to your Databricks account team.

• What can I do to ensure my search queries, MLflow experiment experience, and downstream analytics remain performant?

  Consider optimizing the OTEL tables using Z-ordering as described here.

• How does this handle PII found in user prompts?

  This feature does not apply any special handling to PII. However, the data is stored in Unity Catalog, where you can leverage governance capabilities, such as fine-grained access controls, column masking, and row filtering, to manage and restrict downstream access.

Get started

To get started, follow along with the documentation.

Integrate Databricks Genie with Google Chat in 30 mins!

With this integration, your team can query business data directly from Google Chat using natural language. No SQL knowledge needed, no switching between tools.

Features

• Conversational AI in Google Chat - Ask questions like “What were total sales last quarter?” and get instant, data-backed answers

• Rich Card Responses - Results are displayed as formatted Google Chat Cards with data tables, generated SQL

• Response Feedback Loop - Thumbs up/down buttons send ratings back to Genie so space authors can review and improve

• Direct Messages and @Mentions - Works in 1:1 DMs with the bot or by @mentioning it in any Chat Space

• No Infrastructure Required - Runs entirely on Google Apps Script (serverless), no servers to manage, no Databricks App to deploy

• Secure by Default - Credentials stored in Apps Script Properties Service, data governed by Unity Catalog

Architecture

Prerequisites

1. Databricks Workspace

   • Active Databricks workspace

   • Genie space ID

2. Google Workspace with the Enterprise plan

   • Enable Google Chat API in the project

   • Enable App Script in the same project

Setup Instructions

Set up Genie Space in Databricks Workspace and note the few parameters

Genie Details

If you already have a Genie Space, skip to the next step. Otherwise:

1. In your Databricks workspace, navigate to Genie in the left sidebar

2. Click New in the upper-right corner of the screen

3. Add your data sources. Then, click Create

4. From the Settings tab, note the Space ID (a 32-character string) -- you will need this later

Tip: The quality of your Genie Space directly affects the quality of your answers. Add company-specific context, test with expected user questions, and iterate on your table annotations.

Generate a Databricks Token

Apps Script needs a token to authenticate with the Genie API.

1. In your Databricks workspace, click your username (top right) then Settings

2. Go to Developer then Access Tokens

3. Click Generate New Token, give it a description like “Google Chat Genie Bot”

4. Copy the token immediately -- you will not see it again

Also note your workspace URL (e.g., https://your-instance.cloud.databricks.com).

For production: Use OAuth M2M with a service principal instead of a personal access token for better security and automated token rotation.

Google Workspace Setup

Google Cloud Project Setup

1. Go to the Google Cloud Console (console.cloud.google.com)

2. Create a new project (or select an existing one)

3. Navigate to APIs and Services then Library

4. Search for Google Chat API and click Enable

5. Go to APIs and Services then OAuth consent screen

6. Select Internal, fill in the app name (e.g., “Genie Data Bot”), and save

Create the Apps Script Project

1. Go to script.google.com and click New project

2. Rename the project to Genie Chat Bot

3. Download all files from the repo https://github.com/adgitdemo/ad_databricks/tree/main/genie-google-chat-app and create the corresponding .gs files in the Apps Script editor:

• Code.gs -- Chat event handlers (onMessage, onAddedToSpace, submitFeedback)

• appscript.json - App Script configuration

• (Recommended) Go to Project Settings (gear icon) then under the “Script Properties“ section, click Add script property (or Edit script properties):

  • DATABRICKS_TOKEN = your token from Step 1

  • GENIE_SPACE_ID = your space ID from Step 1

  • DATABRICKS_HOST = your workspace URL (no trailing slash)

• The code reads from Script Properties automatically.

Configure the Google Chat API

1. In Apps Script, click Deploy->New Deployment, select Add-On configuration, and deploy. Copy the Deployment ID once process completed.

2. Back in the Google Cloud Console, go to APIs and Services then Google Chat API then Configuration

3. Fill in:

- App name: db-genie-1

- Avatar URL: (optional -- your company logo or Databricks icon)

- Description: Databricks Genie Bot

- Functionality: Check “Join spaces and group conversations”

- Connection settings: Select Apps Script and paste your Head Deployment ID in the Deployment ID field

- Triggers specify the apps script functions to handle interactions

- Visibility: Select “Specific people and groups” and add your team or test users

Add Chat App/Bot App to Google Chat

1. Open the Space

Go into the Space where you want the app.

2. Click the Space name at the top

A menu opens.

3. Choose “Manage apps”

Then click Add apps.

4. Search + Add the app

Usage

Once deployed, your team can interact with the bot in two ways:

Direct Messages -- Open a DM with Genie Data Bot and type your question directly. Great for ad-hoc data exploration.

@Mentions in Spaces -- In any Google Chat Space where the bot is added, type @db-genie-1 followed by your question. Responses appear in the space, and anyone can ask follow-ups in the thread.

Demo Time

Tips and Considerations

• Answer quality depends on your Genie Space. Add detailed table/column descriptions, sample SQL, and company-specific context for best results.

• Sync timeout: Apps Script has a 30-second synchronous response limit for Chat. If your Genie queries take longer, consider an optional async pattern using time-driven triggers and the Advanced Chat Service.

• Rate limits: The Genie API allows approximately 5 queries per minute per workspace during Public Preview.

• Row limits: Genie returns up to 5,000 rows per query. The Chat card displays up to 20 rows for readability, with a link to view full results in Genie.

As a Specialist Solutions Architect at Databricks, I regularly work with customers who are migrating critical analytics from existing BI tools to Databricks AI/BI Dashboards – and the first questions I usually get are about filters.

Teams want to know:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

These aren’t cosmetic features. They’re how analysts actually interact with dashboards, and they’re often the reason an existing BI dashboard feels “alive” instead of static.

In this post, I’ll walk through how to implement two familiar filter patterns from existing BI dashboards in Databricks AI/BI Dashboards, using the built-in samples.tpch dataset:

1. Context filters → implemented as parameters in dataset SQL

2. “Only Relevant Values” or cascading filters → implemented with field filters and query-based parameters

Row-level security and user-based filtering deserve their own deep dive, and action-style interactions (cross-filtering and drill-through) could easily fill another post, so I’ll cover those separately.

I’ve also published the companion dashboard, so you can follow along and inspect the configurations yourself.

Quick primer: datasets and filters in Databricks AI/BI Dashboards

Before we map those patterns, it helps to align on a few AI/BI Dashboards concepts:

Datasets

In AI/BI Dashboards, each dashboard has a Data tab where you define one or more datasets:

• A dataset is defined by an SQL query, direct reference to a Unity Catalog table/view, or an uploaded file.

• Multiple visualizations can reuse the same dataset.

• Datasets are bundled with the dashboard when you share/export/import it.

Practically, a dataset is your “model” for a set of visuals: one query, many charts.

Field filters vs parameter filters

AI/BI Dashboards support two core ways to filter data from a dashboard: field filters and parameter filters. Both are implemented as filter widgets, but they behave differently under the hood.

Field filters are applied directly to dataset fields (columns) on top of the dataset query. Processing behaviour is defined by the dataset performance thresholds. Specifically, for small datasets (≤ 100K rows or ≤ 100MB), results are pulled to the browser and visualization-specific filtering and aggregation are applied client-side. For larger datasets, Databricks wraps the dataset query in a WITH clause and applies the filter predicates and aggregations in Databricks SQL warehouse (DBSQL).

Parameter filters are applied to parameters, which are variables that get substituted into your dataset SQL at runtime. When the parameter value changes, the query is always re-run in DBSQL.

In other words, field filters operate on the results of the dataset query, while parameter filters operate inside the dataset SQL itself.

To speed up processing, various caching layers in AI/BI Dashboards and DBSQL are used.

We’ll use parameter filters to emulate context filters, and field filters + query-based parameters to emulate “Only Relevant Values.”

Filter scope: global, page-level, and widget-level

Filters in AI/BI Dashboards also differ by scope:

• Global filters are interactive filters in the global filters panel that apply across all pages of the dashboard to any visualization that shares the selected datasets.

• Page-level filters are interactive filter widgets placed on a specific page in the canvas. They apply to all visualizations on that page that share one or more datasets.

• Widget-level filters are static filters configured directly on a single visualization widget in its configuration panel. Authors set the values, and viewers can’t change them.

With that foundation in place, we can now map these context filters and “Only Relevant Values” patterns into AI/BI Dashboards patterns.

Sample dataset: TPCH on Databricks

To keep examples concrete, we’ll use the TPCH sample data that ships with Databricks in the samples.tpch schema.

For the purposes of this post, you can start by creating a dataset that joins tables region, nation, customer, orders, and lineitem:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

In AI/BI Dashboards, you define this query as a dataset in the Data tab and then reuse it across multiple visualizations. Let’s call this dataset TPCH Sales.

We’ll reuse this same dataset or its derivatives throughout the rest of the post to illustrate context filters and cascading filters.

Implementing context filters with parameters in dataset SQL

What a context filter does

A context filter defines a high-level subset of the data:

• The context filter is applied first, often materializing a temporary subset.

• Other filters and some calculations are then evaluated on top of that subset.

Context filters are used to:

• Improve performance by filtering early and shrinking the working set.

• Enforce logical order, such as “always filter by Region first.”

• Make other filters depend on that subset.

How to think about context in AI/BI Dashboards

Given the primer:

• Field filters operate on the results of the dataset query (Databricks wraps your dataset SQL and applies them on top).

• Parameter filters substitute values directly into your dataset SQL, so they filter inside the query, before joins and aggregations.

If you want “context” behavior – filter first, then apply everything else – you should implement that filter as a parameter in the dataset SQL, driven by a parameter filter widget.

Pattern: treat the context as a base parameter

Let’s add a context filter for Region:

If you’re following along with the companion dashboard, this setup lives on the “Context filter” page.

Step 1. Define TPCH Sales (Context) with a Region parameter

Create a dataset TPCH Sales (Context):

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey
WHERE r.r_name = :region_param      -- “context” filter

In the dataset’s Parameters panel:

• Define region_param with type String.

• Optionally set a default (for example, AMERICA) so the dataset runs without any dashboard filter.

This makes region_param the context for all visuals that use TPCH Sales (Context).

Step 2. (Optional but nice) Create a helper dataset for Region values

You can drive region_param directly from TPCH Sales (Context), but a tiny helper dataset keeps things tidy, convenient, and re-usable.

Create TPCH Regions (Context):

SELECT DISTINCT r_name AS region
FROM samples.tpch.region
ORDER BY region;

This dataset has no parameters; it just returns the list of available regions.

Step 3. Add a Region parameter filter widget

We will configure the widget as a page-level filter (alternatively, you can move it into the global filters panel if it should apply across pages).

On the page where you want Region as a context filter:

1. Add a filter widget and title it Region.

2. Set the filter type to Single value.

3. Configure it as a parameter filter:

   • Fields: TPCH Regions (Context).region

   • Parameters: TPCH Sales (Context).region_param

If you don’t want a helper dataset, you can instead use TPCH Sales (Context).region as the field source, but the wiring is otherwise identical.

Effect

• When a viewer selects Region: EUROPE in the Region filter:

  • The widget writes “EUROPE“ into region_param for TPCH Sales (Context).

  • TPCH Sales (Context) reruns with WHERE r.r_name = ‘EUROPE’.

• All visuals built on TPCH Sales (Context) now use only European data as their starting point.

• Any additional field filters (for example, Nation, Customer, Date) operate on this already-filtered subset, just like secondary filters evaluated after a context filter.

• From the viewer’s perspective, Region behaves like a true context filter: it defines the base subset of data first, and everything else – other filters, cross-filtering, drill-through – is evaluated on top of that context.

Implementing “Only Relevant Values” with cascading filters and query-based parameters

What “Only Relevant Values” behavior means

“Only Relevant Values” behavior on a filter shrinks the list of values based on the current state of other filters and the view:

• If you select Region: ASIA, the Country filter only shows countries that actually have data in ASIA.

• As you add more filters, each filter’s domain is recomputed from the filtered dataset.

Practically, this gives you cascading filters that stay in sync with each other and with the current slice of data.

How to think about “Only Relevant Values” in AI/BI Dashboards

In AI/BI Dashboards, you get the same effect in two ways:

1. Field filters on the same dataset – AI/BI recomputes the value list based on the current filtered dataset.

2. Query-based parameters – a specialized filter widget that both populates its values from a query, and writes the selected value into a parameter used in your dataset SQL.

Pattern 1: Cascading filters with field filters

The simplest way to mimic “Only Relevant Values” is to use field filters wired to the same dataset. AI/BI Dashboards will automatically recompute each filter’s value list based on the current filtered dataset.

We’ll build a Region → Nation → Customer cascade on top of TPCH Sales (Cascading Pattern 1).

In the companion dashboard, this pattern is implemented on the “Cascading filters with field filters” page.

Step 1. Define TPCH Sales (Cascading Pattern 1)

Create a dataset TPCH Sales (Cascading Pattern 1) with the base TPCH join and a revenue metric:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

This dataset has no parameters; all filtering will be done with field filters on top of the query results.

Step 2. Add Region, Nation, and Customer field filters

On the dashboard page where you want cascading behavior:

1. Add three field filter widgets with titles Region, Nation, and Customer.

2. Configure each widget as a page-level filter (or move them into the global filters panel if you want them to apply across pages).

3. Connect the filters to the following fields from TPCH Sales (Cascading Pattern 1):

   • Region → region

   • Nation → nation

   • Customer → customer_id

No parameters are involved here – these are pure field filters on a single dataset.

Effect

• When a viewer selects region: ASIA, the TPCH Sales (Cascading Pattern 1) dataset is filtered to ASIA for all visuals on the page.

• The Nation field filter’s value list is recomputed from that filtered dataset, so it only shows nations in ASIA.

• After the viewer chooses a nation (e.g., JAPAN), the Customer field filter shrinks to show only customers in that nation.

• From the user’s perspective, these field filters behave like filters with an “Only Relevant Values” option enabled: each dropdown shows only values that exist in the currently filtered data. Under the hood, AI/BI Dashboards are simply applying field filters on top of a single dataset and recomputing the dropdown values from the currently filtered result set.

Pattern 2: Cascading filters with query-based parameters

In Pattern 1, we used field filters only. In some cases you may want more control over how dropdown values are loaded, or you may want the same parameter to drive multiple datasets. In that case you can use query-based parameters. A query-based parameter filter widget gets its dropdown values from a field in a “choices” dataset, and writes the selected value into one or more parameters that are used in dataset SQL.

Here we’ll build a three-level cascade Region → Nation → Customer using:

• One main dataset: TPCH Sales (Cascading Pattern 2)

• Three small “value list” datasets:

  • TPCH Regions (Cascading Pattern 2)

  • TPCH Nations by Region (Cascading Pattern 2)

  • TPCH Customers by Nation (Cascading Pattern 2)

In the companion dashboard, this pattern is implemented on the “Cascading filters with query-based parameters” page.

Step 1. Define TPCH Sales (Cascading Pattern 2)

Create the TPCH Sales (Cascading Pattern 2) dataset with parameters for region, nation, and customer:

SELECT
 r.r_name              AS region,
 n.n_name              AS nation,
 c.c_custkey           AS customer_id,
 c.c_name              AS customer_name,
 o.o_orderkey          AS order_id,
 o.o_orderdate         AS order_date,
 l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey
WHERE (:region_param   = 'All' OR r.r_name = :region_param)
  AND (:nation_param   = 'All' OR n.n_name = :nation_param)
  AND (:customer_param = 0     OR c.c_custkey  = :customer_param);

In the dataset’s Parameters panel:

• Set region_param type to String.

• Set nation_param type to String.

• Set customer_param type to Numeric / Integer (to match c_custkey).

This last bit is important: the Customer filter uses a numeric field, so the parameter must be numeric as well.

Step 2. Create helper datasets for the dropdowns

1. TPCH Regions (Cascading Pattern 2) – list of regions:

SELECT DISTINCT r_name AS region
FROM samples.tpch.region
ORDER BY region;

2. TPCH Nations by Region (Cascading Pattern 2) – nations for the selected region:

SELECT DISTINCT n.n_name AS nation
FROM samples.tpch.nation   AS n
JOIN samples.tpch.region   AS r ON n.n_regionkey = r.r_regionkey
WHERE r.r_name = :region_param
ORDER BY nation;

This dataset defines its own region_param (string) in the Data tab.

3. TPCH Customers by Nation (Cascading Pattern 2) – customers for the selected nation:

SELECT DISTINCT
  c.c_custkey AS customer_id
FROM samples.tpch.nation   AS n
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
WHERE n.n_name = :nation_param
ORDER BY customer_id;

This dataset defines nation_param (string). customer_id is numeric, matching customer_param in TPCH Sales (Cascading Pattern 2).

Run each dataset in the Data tab once to confirm they succeed.

Step 3. Add Region, Nation, and Customer filter widgets

On your dashboard page, add three filter widgets and wire them to fields and parameters. Configure all three widgets as page-level filters (or move them into the global filters panel if they should apply across pages).

1. Region filter widget

• Filter type: Single value

• Fields: TPCH Regions (Cascading Pattern 2).region

• Parameters:

  • TPCH Sales (Cascading Pattern 2).region_param

  • TPCH Nations by Region (Cascading Pattern 2).region_param

• Default value: All

This keeps region_param in TPCH Sales (Cascading Pattern 2) and TPCH Nations by Region (Cascading Pattern 2) in sync.

2. Nation filter widget

• Filter type: Single value

• Fields: TPCH Nations by Region (Cascading Pattern 2).nation

• Parameters:

  • TPCH Sales (Cascading Pattern 2).nation_param

  • TPCH Customers by Nation (Cascading Pattern 2).nation_param

• Default value: All

This keeps nation_param in TPCH Sales (Cascading Pattern 2) and TPCH Customers by Nation (Cascading Pattern 2) in sync.

3. Customer filter widget

• Filter type: Single value

• Fields: TPCH Customers by Nation (Cascading Pattern 2).customer_id

• Parameters: TPCH Sales (Cascading Pattern 2).customer_param

• Default value: 0

Effect

• When a viewer selects Region: AMERICA:

  • The Region widget writes “AMERICA“ into region_param in TPCH Sales (Cascading Pattern 2) and TPCH Nations by Region (Cascading Pattern 2).

  • TPCH Nations by Region (Cascading Pattern 2) reruns and returns only nations in AMERICA, so the Nation dropdown only shows those nations.

• When the viewer then selects Nation: UNITED STATES:

  • The Nation widget writes “UNITED STATES“ into nation_param in TPCH Sales (Cascading Pattern 2) and TPCH Customers by Nation (Cascading Pattern 2).

  • TPCH Customers by Nation (Cascading Pattern 2) reruns and returns only customers in UNITED STATES, so the Customer dropdown only shows those customer IDs.

• When the viewer selects a specific Customer (for example, 607):

  • The Customer widget writes 607 into customer_param in TPCH Sales (Cascading Pattern 2).

  • TPCH Sales (Cascading Pattern 2) reruns with region_param, nation_param, and customer_param applied, and all visuals built on this dataset show only orders for customer 607 in UNITED STATES / AMERICA.

• From the viewer’s perspective, Region → Nation → Customer behaves like cascading filters with “Only Relevant Values” behavior enabled. Under the hood, each dropdown is a query-based parameter filter, and the Region and Nation widgets keep parameters in multiple datasets in sync, while Customer filters the main TPCH Sales (Cascading Pattern 2) dataset down to a single customer.

Which pattern when?

Both patterns get you “Only Relevant Values”-style cascading behavior, but they shine in different situations.

Pattern 1 – Cascading field filters

Use this when:

• You’re working off one main dataset per page.

• You want the simplest authoring experience: add field filters, connect them to the dataset, done.

• “Allow All” and easy clearing of filters are important to your users.

This is the closest to what many BI tools do by default and is usually the right starting point.

Pattern 2 – Cascading query-based parameters

Use this when:

• You need parameters that drive multiple datasets.

• You want tighter control over dropdown values, including custom queries per level.

• You’re comfortable managing parameter types and wiring filters to multiple datasets.

Pattern 2 is more flexible and explicit, but also more advanced. In practice, I start with Pattern 1 for most dashboards, and reach for Pattern 2 when I need parameter-driven logic or want to reuse the same parameters across several datasets and pages.

Summary

In this post, we looked at how to carry two of the most important filter patterns from traditional BI dashboards into Databricks AI/BI Dashboards:

• Context filters become parameters in your dataset SQL, driven by parameter filter widgets. This lets you enforce “filter by Region first” semantics and shrink the working set before joins and aggregations.

• “Only Relevant Values” / cascading filters can be implemented either with simple field filters on a single dataset (Pattern 1) or with query-based parameters and helper datasets (Pattern 2) when you need more control and reusable parameters.

The companion dashboard includes all three examples: a context filter page, a field-based cascading page, and a query-based cascading page. You can import it into your workspace and adapt the patterns to your own datasets.

In future posts, I plan to cover:

• Row-level security and user-based filtering in AI/BI Dashboards

• Action-style interactions such as cross-filtering and drill-through in AI/BI Dashboards

If you’re starting a migration from an existing BI tool to Databricks AI/BI today, I recommend:

1. Identify your key context filters (Region, Business Unit, etc.) and implement them as parameters in dataset SQL.

2. Start with Pattern 1 (field filters) for cascading behavior, and only move to Pattern 2 where you truly need parameter-driven logic or shared parameters across datasets.

These two patterns alone are usually enough to make an AI/BI dashboard feel as interactive and “alive” as the dashboards your teams are used to today.

The goal was simple but demanding: near-real-time data availability and consistently fast query performance across large time ranges, even in the presence of late-arriving data and massive daily ingestion volumes.

Why Traditional Partitioning Fell Short and How Liquid Clustering Solves It

The original architecture relied on continuous streaming ingestion into Bronze tables from Kafka, followed by scheduled batch jobs that populated optimized Silver tables.Bronze tables were partitioned by date, while Silver tables were partitioned and z-ordered by relevant keys. When everything arrived on time and tables were fully optimized, query performance was excellent.

The problem began with late-arriving data.

Some events arrived days—even weeks—after their original event time. These records landed in partitions that had already been optimized, slowly reintroducing small, unoptimized files into previously clean partitions. Over time:

• File counts ballooned

• Queries that once ran in ~10 seconds stretched to a minute or more

• Performance degraded steadily as more late data accumulated

The only way to recover performance was to re-optimize entire partitions repeatedly, which became increasingly expensive and time-consuming at scale. Rigid partition boundaries simply did not work well with unpredictable arrival patterns.

This is where Liquid Clustering became a natural fit. Instead of relying on static partitions, Liquid Clustering incrementally maintains data layout quality as data arrives. It continuously rebalances files based on clustering keys, reducing the need for repeated full rewrites and making late-arriving data far less disruptive.

Liquid Clustering addresses these issues with a multi-dimensional, incremental clustering strategy. It removes rigid partition boundaries, continuously reorganizes poorly clustered segments, and supports both eager clustering (during ingestion) and lazy clustering (post-ingestion). Using a tree-based multi-column clustering, it improves data skipping and maintains predictable, low-latency query performance even on large, late-arriving datasets.

Where Liquid Met Production Reality: Scaling Challenges and Fixes

While Liquid Clustering addressed the core architectural issue, the migration itself surfaced a new set of challenges primarily driven by scale.

High-Throughput Ingestion and Backfills

One of the first challenges during the migration to Liquid Clustering was handling the existing historical data. To adopt the new layout strategy, tens of terabytes of data per day over several months had to be backfilled and reorganized. This was challenging because the platform also had to continue processing new streaming data and maintain Silver tables for analysts. Running backfill and OPTIMIZE jobs simultaneously put the system under extreme load, pushing cluster resources to their limits.

Long-Running OPTIMIZE Jobs After Enabling Liquid

After enabling Liquid Clustering, OPTIMIZE runtimes increased noticeably—not because of Liquid alone, but due to the scale at which it was introduced. Large historical backfills were running alongside ongoing ingestion, forcing OPTIMIZE to process very large data volumes under heavy skew. Certain clustering stages ended up handling disproportionate amounts of data, resulting in large shuffles, disk spills, and increasingly unpredictable runtimes.

To address this, eager clustering was enabled for streaming writes, moving a portion of the clustering work into ingestion. This reduced the amount of reorganization required during OPTIMIZE and helped stabilize optimization runtimes, especially once batch sizes were increased and clustering work was better distributed across the cluster.

In addition, Liquid-specific tuning played a critical role in improving OPTIMIZE stability at scale. Key adjustments included:

• Enhanced data skipping was enabled to reduce unnecessary data movement during clustering, significantly lowering shuffle volume and improving OPTIMIZE efficiency.

• Increased clustering parallelism was also configured to distribute clustering work more evenly across the cluster, reducing skew and stabilizing runtimes for large and wide tables.

For workload-specific tuning and the exact configuration details, we recommend reaching out to Databricks, as the optimal settings can vary based on data volume and cluster characteristics.

To further reduce reliance on manual OPTIMIZE jobs, we also leveraged Predictive Optimization (PO) to automatically manage optimization workloads—this topic is discussed in more detail in the upcoming section.

Eager Clustering at Scale: Small File Challenges

While eager clustering reduced the amount of work during OPTIMIZE, it introduced a new challenge when streaming batches were too small. This was especially noticeable during large historical backfills, where each batch was around 40 GB. Although each batch was locally clustered, the small size meant that OPTIMIZE still had to rewrite many small files, resulting in high write amplification, longer runtimes, and increased operational overhead.

Solution: Batch Size as a Critical Lever

One of the most important lessons from this migration was how sensitive eager clustering is to batch size, particularly for backfills. Increasing batch sizes to larger, more meaningful units—around 1 TB per batch—changed the behavior dramatically (particularly during petabyte-scale backfills) . Larger batches allowed eager clustering to produce larger, better-clustered files upfront, which significantly reduced or even eliminated downstream OPTIMIZE work. This not only shortened OPTIMIZE runtimes but also lowered overall operational overhead by triggering fewer jobs and improving system stability during high-volume backfill processing.

Infrastructure Constraints

Another challenge emerged from the existing cluster configuration. At petabyte-scale, OPTIMIZE planning occasionally failed due to driver disk exhaustion, caused by large Spark event logs generated during complex optimization planning. The original cluster setup, designed for partitioned tables, was no longer sufficient to handle the heavy resource demands of Liquid Clustering and large backfills.

Solution: Reducing Event Log Volume

The issue was mitigated by tuning Spark to limit event log growth during OPTIMIZE planning. This significantly reduced the size of driver-side logs, preventing disk exhaustion and allowing large optimization jobs to complete reliably even while ingestion workloads continued to run.

For workload-specific tuning and exact configuration details, we recommend reaching out to Databricks.

Cluster and Runtime Constraints

The existing cluster configuration was no longer sufficient to handle simultaneous high-volume ingestion and large-scale OPTIMIZE jobs. Under petabyte-scale workloads, resource contention could slow down processing and introduce instability.

Solution: Right-Sizing Compute and Updating Runtime

To address this, cluster capacity was increased where needed, reducing reliance on spot instances and favoring on-demand workers for stability during long-running OPTIMIZE operations. In addition, the latest DBR 17.3 runtime was adopted for all new Liquid tables, leveraging improvements that enhanced performance, stability, and optimization efficiency at scale.

Cluster capacity was adjusted to reduce reliance on spot instances for long-running OPTIMIZE jobs, favoring on-demand workers where stability mattered most.

Predictive Optimization: Powerful, but Needs Guardrails

Once we stabilized OPTIMIZE runtimes and tuned eager clustering, the next focus was on reducing reliance on manual optimization jobs. For this, we leveraged Predictive Optimization , which automatically determines when and how to run clustering operations based on table state and data layout.

Concurrent manual OPTIMIZE jobs sometimes conflicted with PO runs in a few cases , causing transaction failures. Limited observability made it difficult to track PO activity and determine how much data remained unoptimized, which could lead to degraded query performance.

To address this, we implemented several best practices:

• Monitor PO activity using system.storage.predictive_optimization_history to track execution and outcomes.

• Fallback manual OPTIMIZE jobs are run whenever PO does not execute successfully.

Results: Performance, Freshness, and Simpler Operations

The migration to Liquid Clustering delivered clear, measurable improvements:

• Query performance improved dramatically—around 50% faster for most queries and up to 90% faster for long-range scans

• File counts were cut roughly in half, reducing I/O overhead

• Data freshness improved from hours to minutes, enabling near-real-time analytics

• Operational overhead dropped significantly with UC-managed tables and automated optimization

• Legacy partitioning was eliminated, reducing technical debt and modernizing the architecture
  

Final Thoughts

Liquid Clustering fundamentally changes how data layout is managed at scale. By moving away from rigid partitions and embracing incremental, adaptive clustering, it becomes possible to handle late-arriving data, massive ingestion volumes, and evolving schemas without sacrificing performance or driving up costs.

The key is understanding the operational nuances: batch sizing, clustering strategy, Liquid-specific tuning, cluster configuration, and optimization automation. When those pieces come together, Liquid Clustering can unlock faster queries, fresher data, and a far more resilient data platform.

For readers interested in diving deeper:

• Arctic Wolf’s Liquid Clustering Architecture Tuned for Petabyte Scale – Databricks Blog

• Optimizing Delta Lake Tables with Liquid Clustering – Canadian Data Guy

The Knowledge Assistant brick is one of several specialized agent types available in Databricks Agent Bricks, which is a low-code solution to help developers easily create and optimize domain specific agents with a few clicks.

The Knowledge Assistant brick is a no-code RAG agent that delivers reliable responses with citations. Instead of relying solely on what a language model learned during training, RAG returns relevant information from your documents in real time and is used to generate responses. The Knowledge Assistant brick uses optimizations out of the box, so you do not need to worry about optimizing the index or LLM. In our demo, we built a Knowledge Assistant for a team working with Databricks. The agent uses blog posts (written by our team, here on Databricksters) to answer questions about Spark streaming, coding practices, etc.

This same approach can work for any scenario where you have custom knowledge and want to create “general intelligence” on your data. Think of unstructured data that you already have-- HR policies, production information, or technical docs.

Prerequisites

Before building your own brick, ensure your workspace meets these requirements:

• Mosaic AI Agent Bricks Preview (Beta) enabled

• Production monitoring for MLflow (Beta) enabled

• Serverless compute enabled

• Unity Catalog enabled

• Access to Mosaic AI model serving

• Access to foundation models in Unity Catalog through system.ai schema

• Serverless budget policy with non-zero budget

• Workspace in a supported region

Data Requirements

Make sure you have one of the following:

• Files in a Unity Catalog volume (supported formats: txt, pdf, md, ppt, docx)

• A vector search index

  • The databricks-gte-large-en embedding model endpoint must have AI guardrails and rate limits disabled.

Adding feedback and improving Agent Quality through feedback

You can add feedback and assessments to each trace. This is where the human feedback begins-- you can start noting which responses are good, what needs improvement, etc. These assessments become a part of the MLflow experiment associated with the Agent Brick. You can aggregate this information using the MLflow API.

In my opinion, this is where Agent Bricks really shines. The Knowledge Assistant can automatically update itself according to the human feedback using a process called ‘Agent Learning from Human Feedback’. Learn more about this here. This allows the agent to continuously improve based on expert expectations -- without you needing to retrain models or fiddle with prompts manually.

The feedback loop

The process is straightforward: (1) create your challenging questions, (2) have SMEs review the agent responses and add comments, and then (3) sync the feedback. “Syncing” will start the process to improve the agent based on the feedback.

How should you approach writing questions? Similar to creating an evaluation set, you should include a variety of questions that are critical for the use-case. Based on previous evaluations, you can include edge-case questions as well to see what SMEs want to see. Remember: you are only providing the questions, not the answers. The agent will generate the answers, and then the SMEs will provide the feedback on the answers.

Once you have your questions added, you can start a Labeling Session. Once that session is ready, share that link with SMEs and wait for the feedback to roll in. SMEs can add feedback on tone, style, or accuracy of that agent response. Make sure you communicate how to add feedback to your SMEs!

After ending a Labeling Session (after SMEs have added their feedback), you can start syncing the responses. Agent Bricks uses many techniques to sync the feedback with the agent. Congratulations! You have completed the first human feedback loop!

The Problem

Complex Ingestion Architecture

Today’s data teams face a common challenge: streaming data from applications to their Lakehouse requires maintaining complex infrastructure. The typical setup involves managing a message bus like Kafka, configuring connectors, monitoring pipelines, and dealing with significant operational overhead and costs—all just to move data from point A to point B.

Managed Bus Don’t Solve Everything

While Amazon Managed Streaming for Apache Kafka(Amazon MSK) removes the burden of managing servers, it doesn’t eliminate your responsibility for the message bus itself. You’re still on the hook for capacity planning, topic and partition design, producer and consumer tuning, monitoring and alerting, and upgrade timing. Managed services make these tasks less manual, but upgrades remain risky and the operational complexity persists.

Cost management is another pain point. Amazon Managed Streaming for Apache Kafka(Amazon MSK) bills can balloon quickly due to over-provisioned brokers, excess partitions, high replication factors, and long retention periods. AWS manages the infrastructure, but not your spending discipline—that’s still your problem.

What You Actually Need

A fully abstracted streaming service that eliminates cluster management entirely, letting you focus on building data products instead of babysitting message bus infrastructure.

The Challenge

A rapidly growing automotive startup was processing massive device data volumes from Go applications. After essential first-level processing, they needed to stream data to their data lake for near real-time analytics. To avoid the complexity of managing Kafka or similar message bus infrastructure, they took a shortcut: direct writes to their data warehouse with append-only inserts.

Initially simple, this approach quickly hit walls. As volumes grew, they vertically scaled, then horizontally distributed producers across multiple warehouse instances. Small but relentless queries created network bottlenecks. Excessive delta commits from numerous producers killed throughput. They hit soft limits on connections and write operations. To keep data flowing, they over-provisioned compute—watching costs balloon without proportional gains.

The Solution

Zerobus Ingest provided the purpose-built ingestion layer they needed. Their Go applications integrated the SDK with minimal code changes—same append-only pattern, properly architected. The Write-Ahead Logging (WAL) based system handled buffering and batching, while automatic recovery managed network issues that previously caused data loss.

Data now lands directly in Delta tables, eliminating the warehouse intermediary. The result: lower latency, higher throughput, dramatically reduced costs, and one less system to manage. They got the simplicity of direct writes with the scalability of proper streaming—at a fraction of Kafka’s cost.

Overview

Zerobus Ingest is a fully managed, zero-configuration service that enables record-by-record data ingestion directly into Delta tables. No more intermediate message buses. No more complex configurations. Just point your application at an endpoint and start sending data. The Zerobus Ingest API buffers transmitted data before adding it to a Delta table. This buffering creates an efficient and durable ingestion mechanism that supports a high volume of clients with variable throughput.

Before:

After:

Features

Zerobus Ingest leverages a Write Ahead Log (WAL) architecture that enables it to store and acknowledge accepted records quickly, delivering low write latency for your applications. The system is backed by persistent disk storage where both the Write Ahead Log (WAL) and checkpoints are maintained, enabling several powerful capabilities:

Automatic Recovery - Network issues are handled transparently by the SDK. It automatically reconnects on transient failures and resends unacknowledged records without requiring any application-level error handling code.

Efficient Resource Management - Once data syncs successfully to Delta tables, Zerobus Ingest automatically cleans up Write Ahead Log (WAL) logs and metadata, freeing disk space for new data without manual intervention.

Schema Management - Automatic validation against your Delta table schema catches data quality issues at ingestion time, preventing malformed data from entering your Lakehouse.

Performance benchmark

Maximum throughput can be achieved when a client app and endpoint are in the same geo region.

• 100MB/second per stream (benchmarked with 1KB-sized messages)

• 15,000 rows per second per stream

SDKs

Users will interact with Zerobus Ingest through a dedicated SDK for their language of choice. The documentation and samples are out for Python SDK, Rust SDK and Java SDK . Both the Go and TypeScript SDKs for Zerobus Ingest are now publicly available. GRPC is the main communication mechanism for Zerobus Ingest.

Databricks documentation contains a well documented guide with sample clients in multiple languages. It guides you right from installing the SDK in your preferred language to creating a Protobuf definition and a sample usage.

Supported Formats

• Protocols: gRPC (primary), HTTP REST, Kafka wire format (coming soon)

• Data Formats: Protocol Buffers, JSON

TIPs

• Visit the table history on UC to get a sense of how frequently the table is updated

• Handle the two exceptions gracefully NonRetriableException, ZerobusException.

• Even though Zerobus Ingest periodically issues data file compactions, so you don’t need to worry about the small files

• Don’t forget to create a table with appropriate data types before you run the client

Components

1. Zerobus Ingest Server - Think of them as scalable stateful pod on K8s attached with an SSD disk(high IOPS). Its responsibilities include:

   • Schema validation of the message to the table.

   • Materializing the data in a timely manner to the target table.

   • Sending an acknowledgement to the client that the data is durable.

2. Smart Networking and orchestration - API proxy which distributes the streams to Zerobus Ingest servers per the target delta table and scales pods as the utilization nears the roof

3. Delta kernel - Record batch writer kicks off every 1-5 seconds, uses Delta kernel(uses Arrow) and writes the record batch to the delta table. Kicks in the PO compaction to avoid small files. Rust APIs hides all the complex details of the Delta protocol specification. Binding available for python.

4. Write-Ahead Log(WAL) - Records are immediately persisted to durable storage(think SSD disks with high IOPS) provided by the cloud platform your databricks is running on and is acknowledged in under 50ms. This guarantees durability even if something fails

Unofficial Zerobus Ingest overview

Zerobus Ingest fits direct lakehouse writes with durable acknowledgments (no bus-style retention/multi-consumer). Zerobus Ingest it not a replacement of message bus in all scenarios. If you need message bus durability/retention or multiple subscribers, Event Hubs/Kafka is likely a safer choice.

Availability

Databricks only support single availability zone (single AZ) durability. This means Zerobus Ingest service may experience downtime. This might change soon.

Kafka still wins when

Despite the cost advantages of Zerobus Ingest Ingest, Kafka remains a better choice in following scenarios:

Exactly-once semantics requirements - For financial transactions, order processing, or other workflows where duplicate processing could cause serious issues, Kafka’s exactly-once delivery guarantees are critical. While Zerobus Ingest roadmap includes this feature, organizations that need it today must still rely on Kafka.

Ultra-low latency fan-out - If your use case requires multiple consumers reading the same stream with different processing logic, Kafka’s pub-sub model excels. Zerobus Ingest currently lacks the subscriber/consumer model that makes Kafka so powerful for fan-out patterns where one stream feeds multiple downstream applications.

Other Limitations

As of writing,

• Zerobus Ingest provides at-least-once delivery semantics, meaning each message will be delivered one or more times. It does not yet support exactly-once semantics. However, the duplicates can be handled using other Databricks and delta features.

• Zerobus Ingest currently supports writing only to managed Delta tables

• Schema evolution on target tables is not yet supported in Zerobus Ingest, so the table schema must match the incoming message structure.

• Each individual message is limited to a maximum size of 10 MB when processed through Zerobus Ingest.

What’s Next

Databricks is actively enhancing Zerobus Ingest with several key features in development. The roadmap includes exactly-once delivery semantics for stronger consistency guarantees, MQTT protocol support to broaden IoT and device connectivity options, comprehensive CDC pipeline capabilities that will handle updates and deletes in addition to inserts and subscriber/consumer model to enable more flexible data consumption patterns.

Enjoy streaming in a cost efficient and simplified manner!

Zerobus Ingest offers a compelling alternative to message bus in a lot of scenarios. While Kafka remains essential for complex streaming architectures, Zerobus Ingest closes the gap for straightforward ingestion use cases—delivering the reliability you need at a fraction of the cost and complexity.

The cost savings extend beyond infrastructure. Kafka expertise commands premium salaries, and maintaining distributed message bus systems requires dedicated engineering time that could be spent on higher-value work. Zerobus Ingest’s simplicity means junior engineers can manage what previously required highly skilled distributed systems expertise. When you factor in reduced operational overhead, lower training costs, and faster time-to-production, the economics become even more compelling. Sometimes the best architecture isn’t the most sophisticated—it’s the one that solves your problem.

I’ve written an example custom Salesforce batch writer here. It is designed to support uploads via the Salesforce Bulk API v1.0 or v2.0, but this guide will focus on v2.0. I strongly recommend using 2.0 for newly introduced features discussed below.

This code is a robust example, and is not intended to be copy/pasted into production.

Background on Salesforce Bulk API 2.0

Before diving into the code, let’s discuss how the Bulk API 2.0 works and how our writer can interact with it. A “job” is the unit of work in the Salesforce Bulk API. In v1 of the Salesforce API, users had to create a job and then manually break that job into chunks of 10,000 records and track each batch individually. In v2, you simply submit your data via a job and it handles batches and retries automatically. The v1 limit for upload was 10MB per batch, while the v2 limit is 150MB per job.

But what if my DataFrame is larger than 150MB, do I have to manually slice the data into sub 150MB chunks and iteratively submit multiple Bulk API jobs? This is where the power of Spark shines, though it comes with a requirement. Spark will automatically parallelize the work into multiple Salesforce “jobs”, but it won’t automatically slice large partitions. You must explicitly control the partition size (using repartition) to ensure you don’t send a chunk larger than 150MB.

Background on Spark writes

In Spark, when df.write is called, the driver calls the writer() method for the DataSource object in question. In our Salesforce example, calling this writer method will result in the instantiation of our SalesforceBatchWriter object.

def writer(self, schema: StructType, overwrite: bool):
    “”“Create a writer instance for the given schema.”“”
    return SalesforceBatchWriter(self.options, schema)

class SalesforceBatchWriter(DataSourceWriter):
    “”“
    DataSourceWriter implementation for Salesforce Bulk API operations.
    Handles both authentication and bulk data upload to Salesforce objects.
    “”“
    
    def __init__(self, options: Dict[str, str], schema: StructType):
        self.options = options
        self.schema = schema

Spark then looks at the write() method within your DataSourceWriter (recall above, our SalesforceBatchWriter inherited from the DataSourceWriter class), serializes (pickles) this write method, and creates a task for every partition in your DataFrame. It then sends these tasks to the Executors. Simply put, in Spark, each partition of data receives its own write task. This means if we have an extremely large DataFrame, so long as its partitioned and each partition is under 150MB, each partition will receive its own set of write instructions and can be submitted in parallel as multiple Salesforce Bulk API v2 jobs.

Spark to Salesforce Writer

Now that we understand a bit more about how Spark and the Salesforce Bulk API can interact, let’s dive into our custom writer implementation.

def write(self, rows: Iterator[Row]) -> SalesforceCommitMessage:
        “”“
        Write rows to Salesforce using Bulk API.
        
        Args:
            rows: Iterator of PySpark Row objects to write
            
        Returns:
            SalesforceCommitMessage with write statistics
        “”“
        # Import inside method to meet serialization requirements on executors
        from simple_salesforce import Salesforce, SalesforceAuthenticationFailed

        ctx = TaskContext.get()
        partition_id = ctx.partitionId()

        username = self.options.get(”username”)
        password = self.options.get(”password”)
        security_token = self.options.get(”security_token”)
        sobject = self.options.get(”sobject”)
        instance_url = (self.options.get(”instance_url”) or “”).strip()
        domain = (self.options.get(”domain”) or “login”).strip()
        api_version = self.options.get(”api_version”, “1”)  # “1” (Bulk V1) or “2” (Bulk V2)

        if not all([username, password, security_token, sobject]):
            raise ValueError(”Missing required Salesforce options: ‘username’, ‘password’, ‘security_token’, ‘sobject’”)

        # Collect iterator to a list of dicts for bulk insert
        data_to_upload: List[Dict[str, Any]] = [row.asDict() for row in rows]

        if not data_to_upload:
            print(f”Partition {partition_id}: No rows to write.”)
            return SalesforceCommitMessage(partition_id=partition_id, records_written=0, errors=0)

        try:
            if instance_url:
                sf = Salesforce(
                    username=username,
                    password=password,
                    security_token=security_token,
                    instance_url=instance_url,
                )
            else:
                sf = Salesforce(
                    username=username,
                    password=password,
                    security_token=security_token,
                    domain=domain,
                )

The data_to_upload is an important array variable in this writer. To understand what it is doing, recall that write() is pickled and sent as a task for each partition in your DataFrame. This means that the rows: Iterator[Row] argument that data_to_upload is iterating over is the set of rows for one partition. This is turning each row into a key/value dict for compatibility with the Salesforce API and storing them in a Python List. It’s extremely important to recognize that creating this List of Dict objects is materializing the entire partition in the memory of your executor. This must be done, as the Salesforce API requires the full payload to be constructed before sending. If this is done on a partition that is too large, you will face Out of Memory (OOM) issues. If this is the case, logic could be added to chunk your partition, but that would mean additional Bulk API jobs and additional API calls in Salesforce.

The next step is to actually perform the bulk insert of these records and track the status of the job returned by the API.

if api_version == “2”:
                job_summaries = self._bulk2_insert(sf, sobject, data_to_upload, batch_size)
                
                for summary in job_summaries:
                    num_processed = int(summary.get(”numberRecordsProcessed”, 0))
                    num_failed = int(summary.get(”numberRecordsFailed”, 0))
                    success += max(0, num_processed - num_failed)
                    errors += max(0, num_failed)

                    print(f”Partition {partition_id} job {summary.get(’job_id’)} summary: “
                        f”processed={num_processed}, failed={num_failed}, total={summary.get(’numberRecordsTotal’)}”)

                    if num_failed > 0 and summary.get(”job_id”):
                        try:
                            failed_csv = sf.bulk2.__getattr__(sobject).get_failed_records(summary[”job_id”])
                            if isinstance(failed_csv, str):
                                preview = failed_csv[:2048]
                                print(f”Partition {partition_id} job {summary[’job_id’]} failed records (preview):\n”
                                    f”{preview}”)
                                
                                # Store the first error preview to bubble up to the driver
                                if not failed_records_preview:
                                    failed_records_preview = preview
                        except Exception as e:
                            print(f”Partition {partition_id}: Could not retrieve failed records: {e}”)

job_summaries performs the insert via the _bulk2_call method, and will contain job_status information for the submitted job once it completes. _bulk2_call is an internal method that uses the simple-salesforce insert method and attempts to upload the data in the v2 syntax “insert(records=records)” or if that fails attempts the v1 syntax “insert(data=records)”. It also contains logic to allow for performing upserts instead of inserts.

def _bulk2_call(
        self,
        sf,
        sobject: str,
        operation: str,
        records: List[Dict[str, Any]],
        batch_size: int,
        upsert_key: Optional[str] = None,
    ):
        “”“Execute a Bulk API 2.0 job for the given operation and records.

        Uses the modern signature if available and falls back to legacy signatures.

        Args:
            sf: An authenticated simple_salesforce Salesforce client instance.
            sobject: The target Salesforce object API name (e.g., “Contact”).
            operation: The CRUD operation (”insert”, “update”, “upsert”, “delete”).
            records: The list of prepared record dicts to send.
            batch_size: Preferred batch size hint for the Bulk API 2.0 client.
            upsert_key: External ID field name for upsert operations (required for upsert).

        Returns:
            Any: The result object or list of result objects from the Bulk API call.

        Raises:
            ValueError: If upsert_key is missing for an upsert operation.
        “”“
        api = sf.bulk2.__getattr__(sobject)
        fn = getattr(api, operation)

        try:
            # Prefer modern signature
            if operation == “upsert”:
                if not upsert_key:
                    raise ValueError(”Option ‘upsertField’ is required for upsert.”)
                return fn(records=records, external_id_field=upsert_key, batch_size=batch_size)
            return fn(records=records, batch_size=batch_size)
        except TypeError:
            # Fallback for older versions
            if operation == “upsert”:
                if not upsert_key:
                    raise ValueError(”Option ‘upsertField’ is required for upsert.”)
                return fn(data=records, external_id_field=upsert_key, batch_size=batch_size)
            return fn(data=records, batch_size=batch_size)

Usage

The idea of a reusable and configurable Salesforce writer is great, but how easy is it to use? To use the example I’ve written, simply copy the SalesforceBatchDataSource class anywhere that makes sense for your project. Import the class and register the DataSource with:

spark.dataSource.register(SalesforceBatchDataSource)

Now you’re ready to use the writer!

If running on Databricks, I recommend using dbutils secrets for secret management of your Salesforce credentials. The following code is an example of how to retrieve those and configure your Salesforce connection. instance_host should be your company’s salesforce URI without the https prefix, e.g. mycompany.my.salesforce.com. domain should be “test” for sandbox development, or any other string value for production.

try:
    SF_USERNAME = dbutils.secrets.get(scope=”neil-salesforce”, key=”username”)
    SF_PASSWORD = dbutils.secrets.get(scope=”neil-salesforce”, key=”password”)
    SF_TOKEN = dbutils.secrets.get(scope=”neil-salesforce”, key=”token”)
    SF_DOMAIN = dbutils.secrets.get(scope=”neil-salesforce”, key=”domain”)

    try:
        SF_INSTANCE_HOST = dbutils.secrets.get(scope=”salesforce”, key=”instance_host”)
        SF_INSTANCE_URL = f”https://{SF_INSTANCE_HOST}”
    except Exception:
        SF_INSTANCE_URL = “https://test.salesforce.com” if SF_DOMAIN == “test” else “https://login.salesforce.com”

    sf_creds = {
        “username”: SF_USERNAME,
        “password”: SF_PASSWORD,
        “security_token”: SF_TOKEN,
        “instance_url”: SF_INSTANCE_URL,
    }
    run_job = True
    print(”Loaded Salesforce credentials from Databricks secrets.”)
    print(f”Using instance_url for auth: {SF_INSTANCE_URL}”)
except Exception as e:
    print(f”Warning: Could not load Databricks secrets. {e}”)
    print(”Provide credentials manually or configure secrets; skipping write.”)

Next, the write can be called on our DataFrame. The SF_SOBJECT will be the name of the Salesforce object you are writing to. In my example, I am using a custom “big” object for testing writes larger than the Salesforce developer limit for traditional objects. Another more realistic example of an SF_SOBJECT might be “Contact” or “Account”. Note that the schema of the DataFrame you attempt to write must match the schema that the Salesforce Bulk API V2 expects for that object.

NUM_PARTITIONS = 32
API_VERSION = “2”
SF_SOBJECT = “spark_perf_test__b”

df_perf = df_to_write.repartition(NUM_PARTITIONS)
    
print(f”DataFrame repartitioned into {NUM_PARTITIONS} partition(s).”)
print(”Schema being sent to Salesforce:”)
df_perf.printSchema()
    
 # --- 4. Run the Write Job ---
print(f”Starting write of {TOTAL_RECORDS:,} records to {SF_SOBJECT} via Bulk API v{API_VERSION}...”)

try:
    (
        df_perf.write
        .format(”salesforce-batch”)
        .mode(”append”)
        .options(**sf_creds)
        .option(”sobject”, SF_SOBJECT)
        .option(”api_version”, API_VERSION)
        .save()
        )

Parent/Child Relationships

A must-have for Salesforce integration is the ability to handle objects with parent/child relationships. For example, when uploading a Contact, the Contact should be tied to an Account. This is possible in our example code by performing an upsert, and submitting the Parent ID with the child record.

df_ready = contacts_df.select(
         “FirstName”,
         “LastName”,
         “Email”,  # Contact upsert key in this example
         col(”AccountExtId”).alias(”Account.Oracle_Id__c”)  # Link Contact -> Account by Account external ID
     )

(df_ready.write
     .format(”salesforce-batch”)
     .mode(”append”)
     .options(**sf_creds)
     .option(”sobject”, “Contact”)
     .option(”api_version”, “2”)
     .option(”operation”, “upsert”)
     .option(”upsertField”, “Email”)   # child field to upsert on. unique key
     # Optional tuning:
     # .option(”ignoreNullValues”, “true”)  # omit nulls instead of clearing
     # .option(”batchSize”, “10000”)
     .save()
    )

Null Values

Above we see the option “ignoreNullValues”. This option determines what we do when not all fields are submitted for an object. If ignoreNullValues is set to True, those empty fields in the payload will not be touched within Salesforce. If ignoreNullValues is set to false, those fields will be overwritten with Null values within Salesforce.

Performance

In the snippet above we set NUM_PARTITIONS equal to 32 and repartition the DataFrame with this value. It’s crucial to consider what this value should be set to. As we discussed above, Spark will create a task for each partition, which means creating a Salesforce Bulk API job per partition. This increases throughput, but also multiplies how many Salesforce API calls you make per pipeline run. Each job submitted will use more than one Salesforce API call as it processes, so pay careful attention to your organization’s Salesforce API limits when developing this solution.

With this in mind, for large uploads to Salesforce this solution does outperform submitting all records as one Salesforce Bulk API job. When submitting 300,000 records in one batch, uploading took 372 seconds. When running after repartitioning to 32 partitions (one for each core in my cluster), the time dropped to 31 seconds.

In my performance testing, 32 partitions was chosen to match the 32 cores in my provisioned cluster. For a production use case I recommend finding a balance between the number of partitions you choose to maximize throughput, and the number of Salesforce API calls you are comfortable making per run.

Conclusion

This was an in-depth explanation of how this batch writer has been implemented, and the thought behind it. The final takeaway should be that if you consider some crucial information from this post, writing multiple objects to Salesforce should be as easy as:

# Write the DataFrame `df_accounts` to the standard Account object (recommended: upsert via External ID)
(
    df_accounts.write
    .format(”salesforce-batch”)
    .mode(”append”)
    .options(**sf_creds)
    .option(”sobject”, “Account”)
    .option(”api_version”, “2”)
    .option(”operation”, “upsert”)
    .option(”upsertField”, ACCOUNT_EXT_ID_FIELD)  # e.g., “Oracle_Id__c” or “AccountNumber”
    .save()
)

# Link the DataFrame `df_contacts` to Accounts by flattening the relationship column
# Assumes df_contacts has: FirstName, LastName, Email, AccountExtId (values like “ORA-1001”)
from pyspark.sql.functions import col

df_contacts_ready = df_contacts.select(
    “FirstName”,
    “LastName”,
    “Email”,  # Upsert key on Contact
    col(”AccountExtId”).alias(f”Account.{ACCOUNT_EXT_ID_FIELD}”)  # e.g., “Account.Oracle_Id__c”
)

# Write the linked Contacts (recommended: upsert via Email)
(
    df_contacts_ready.write
    .format(”salesforce-batch”)
    .mode(”append”)
    .options(**sf_creds)
    .option(”sobject”, “Contact”)
    .option(”api_version”, “2”)
    .option(”operation”, “upsert”)
    .option(”upsertField”, “Email”)  # must be non-null/non-empty; ideally External ID/Unique
    .save()
)

Subscribe now

The code I provide will work with SGC or on demand ML Runtime, but it’s significantly faster in SGC even if you only use a single A10 in both scenarios.

If you use MLR, then start with these pip installs to have all the libraries play nicely with MLR 16.4 LTS, if you’re using SGC you should pin these dependencies in the environments tab:

%pip install unsloth[cu124-torch260]==2025.9.6
%pip install threadpoolctl==3.1.0
%pip install accelerate==1.7.0
%pip install unsloth_zoo==2025.9.8
%restart_python

Next we need to get our base model from unsloth. This works for other tuning frameworks like HuggingFace trl as well, but we’re using Unsloth for this demo because it’s nice and quick, and also my customer was using it. Using 4-bit quantization doesn’t necessarily make sense for your production use case, since I’m going to merge it back into 16-bit later anyway, but if you want to save time and RAM during a demo then have at it.

from unsloth import FastLanguageModel
import torch

# Some changes needed for larger models
model, tokenizer = FastLanguageModel.from_pretrained(
   model_name = “unsloth/llama-3.1-8b”,
   max_seq_length = 2048,
   dtype = torch.bfloat16,
   load_in_4bit = True,   # fastest + lowest memory
)

And some sample data:

# It’s a toy set, replace this with your real data
from datasets import Dataset

data = [
   {”text”: “### Instruction: Say hello politely.\n### Response: Hello! How may I ruin your day?”},
   {”text”: “### Instruction: Explain PEFT.\n### Response: A lightweight way to fine-tune large models on the cheap.”},
   {”text”: “### Instruction: Explain LoRA.\n### Response: LoRA adds small trainable matrices instead of updating full weights.”},
]

dataset = Dataset.from_list(data)

Great, now we can define our tokenizer and model as well as tokenize the dataset above:

# Define our tokenizer and tokenize dataset
def tokenize(example):
   encoding = tokenizer(
       example[”text”],
       truncation=True,
       max_length=1024,
       padding=”max_length”,
   )
   encoding[”labels”] = encoding[”input_ids”].copy()
   return encoding

tokenized_dataset = dataset.map(tokenize)

# Define our model
model = FastLanguageModel.get_peft_model(
   model,
   r = 8,
   lora_alpha = 16,
   lora_dropout = 0.0,
   target_modules = [”q_proj”, “k_proj”, “v_proj”, “o_proj”],
)

Similarly for our training args, the trainer, and then we train the model:

from transformers import TrainingArguments, Trainer

# Define our training arguments, trainer, and train toy model
training_args = TrainingArguments(
   output_dir = “outputs”,
   per_device_train_batch_size = 1,
   gradient_accumulation_steps = 1,
   warmup_steps = 0,
   max_steps = 10,
   learning_rate = 5e-5,
   logging_steps = 5,
   optim = “adamw_torch”,
   bf16 = True,
   remove_unused_columns = False,
)

trainer = Trainer(
   model = model,
   args = training_args,
   train_dataset = tokenized_dataset,
)

trainer.train()

Great, so now still only in our VM we’re going to merge this adapter later back into our base weights as promised, though in a real use case you would probably have used 16-bit all along. Here is where you’re really going to see a huge difference between SGC (which does this in ~1 minute) and MLR (which takes ~6):

import shutil
import os

# If you run multiple within short succession, just rename this
LOCAL_TEMP_PATH = “/tmp/llama_merged_model_6”

# See, I told you we would merge back into 16bit weights
model.save_pretrained_merged(
   LOCAL_TEMP_PATH,
   tokenizer=tokenizer,
   save_method=”merged_16bit”,
   safe_serialization=True  # Force safetensors
)

print(”Merged model saved at:”, LOCAL_TEMP_PATH)

Here’s our first stumbling block we’re going to daintily step over. If you don’t do this manually, then your _name_or_path param in your config.json Bourne file will either be set to your base model or won’t be defined at all. In either case, it’s enough for the Provisioned Throughput (PT) endpoint to reject the entire thing:

import json
import os

config_path = os.path.join(LOCAL_TEMP_PATH, “config.json”)

with open(config_path, “r”) as f:
   config = json.load(f)

# Rename the model name to avoid triggering the security check fail, you MUST do this
config[”_name_or_path”] = “unsloth/Meta-Llama-3.1-8B”

# Save the sanitized config back
with open(config_path, “w”) as f:
   json.dump(config, f, indent=2)

print(f”Sanitized config saved to {config_path}”)

Now we’re finally going to save what we have out to a UC Volume, so pick a path you have read and write permissions on:

import subprocess
import mlflow

# Define your paths
UC_VOLUME_PATH = “/Volumes/<catalog_name>/<schema_name>/<volume_name>/merged_weights_mlr_bf16”

# Define the final model name; this must match what the PT Endpoint expects
CATALOG = “<catalog_name>”
SCHEMA = “<schema_name>”
REGISTERED_NAME = f”{CATALOG}.{SCHEMA}.llama_3_1_8b_custom”

# Copy artifacts to volume
print(f”Copying from {LOCAL_TEMP_PATH} to {UC_VOLUME_PATH}...”)

if os.path.exists(UC_VOLUME_PATH):
   subprocess.run([’rm’, ‘-rf’, UC_VOLUME_PATH], check=True)

os.makedirs(UC_VOLUME_PATH, exist_ok=True)

# Copy files
subprocess.run([”cp”, “-r”, f”{LOCAL_TEMP_PATH}/.”, UC_VOLUME_PATH], check=True)
print(”Model copied to UC Volume.”)

Cool, here’s another gotcha: you also need a generation_config.json file to avoid failing the model scan. Here’s one that works; you can change as needed:

# Add generation_config.json to the UC Volume BEFORE logging to MLflow
gen_config = {
   “bos_token_id”: 128000,
   “eos_token_id”: 128001,
   “pad_token_id”: 128004,
   “do_sample”: True,
   “temperature”: 0.6,
   “max_length”: 8192
}

with open(os.path.join(UC_VOLUME_PATH, “generation_config.json”), “w”) as f:
   json.dump(gen_config, f, indent=2)

print(”generation_config.json Bourne added to volume.”)

Alright I can see I’ve overdone the Jason Bourne meme. Lucky for you we’re about done here. We only need to log and register the model to MLflow and then we can serve the fine tuned Llama 3.1 8B model using optimized Provisioned Throughput endpoints for greatly increased token throughput:

# Log and register to MLflow directly from UC Volume path
mlflow.set_registry_uri(”databricks-uc”)

# Define input example for signature (sorry, one more for the road!)
input_example = {
   “messages”: [
       {”role”: “user”, “content”: “That’s Jason Bourne.”}
   ]
}

print(f”Registering model: {REGISTERED_NAME}”)

with mlflow.start_run(run_name=”register_llama_3_1”) as run:
   model_info = mlflow.transformers.log_model(
       transformers_model=UC_VOLUME_PATH,
       artifact_path=”model”,
       task=”llm/v1/chat”,
       input_example=input_example,
       registered_model_name=REGISTERED_NAME,
       metadata={
           “source”: “uc_volume”,
           “original_path”: UC_VOLUME_PATH
       }
   )

print(f”Model version {model_info.registered_model_version} registered.”)

From here you can deploy via the UI or if you want to finish this whole thing in the API you can simply run this with your desired throughput bands. I’m setting mine to the smallest since it’s a demo:

import requests

API_ROOT = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()
API_TOKEN = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()

headers = {
   “Authorization”: f”Bearer {API_TOKEN}”,
   “Content-Type”: “application/json”
}

model_name = “<catalog_name>.<schema_name>.llama_3_1_8b_custom”
model_version = 1 #change as needed
endpoint_name = “llama-31-8b-mlr-test”

payload = {
   “name”: endpoint_name,
   “config”: {
       “served_entities”: [
           {
               “entity_name”: model_name,
               “entity_version”: str(model_version),
               “min_provisioned_throughput”: 19000,
               “max_provisioned_throughput”: 19000,
           }
       ]
   }
}

response = requests.post(
   f”{API_ROOT}/api/2.0/serving-endpoints”,
   headers=headers,
   json=payload
)

print(json.dumps(response.json(), indent=2))

If you navigate to your Serving tab you’ll see this container building. For a toy example on Llama 3.1 8B this should take about 10 minutes.

Please let me know if this unblocked you!

Happy coding.

A common pattern emerges in Delta Lake deployments: storage costs creep up month after month despite stable data ingestion volumes. Teams investigate and discover that their actual in-use data represents only a fraction of their total storage footprint. In one documented case shared on public forums, a team found their active data was just 18% of total storage—the rest was historical data files that had never been cleaned up.

This isn’t a bug—it’s a consequence of how Delta Lake’s time travel feature works. And it’s why the VACUUM command evolved from a basic cleanup tool into a sophisticated optimization toolkit with three distinct modes, each designed for different scenarios.

Understanding the Root Cause: Why Files Accumulate

Delta Lake’s transaction log is the key to understanding storage bloat. Every write operation—INSERT, UPDATE, DELETE, MERGE—generates new Parquet files and records them in the transaction log. But here’s the critical detail: old files aren’t automatically deleted. They’re marked as “removed” in the log, but they remain on disk.

This design enables Delta Lake’s powerful time travel queries:

SELECT * FROM events VERSION AS OF 100
SELECT * FROM events TIMESTAMP AS OF ‘2024-11-01’

The transaction log itself is self-cleaning—log entries older than 30 days (configurable via delta.logRetentionDuration) are automatically pruned. However, the actual Parquet data files are never automatically deleted. Without intervention, they accumulate indefinitely.

For tables with frequent MERGE or UPDATE operations, this can mean months of historical files building up despite a 7-day retention policy.

VACUUM FULL: The Comprehensive Cleanup

The original VACUUM command, now referred to as VACUUM FULL (though FULL is the default and doesn’t need to be specified), was designed as a thorough cleanup mechanism:

VACUUM events RETAIN 168 HOURS

How VACUUM FULL Works

The operation proceeds in three distinct phases:

Phase 1: Recursive File Listing

VACUUM starts by recursively listing every file in the table’s storage directory. On cloud storage, this means API calls:

• AWS S3: ListObjectsV2 requests

• Azure Blob: List Blobs operations

• GCP Storage: List operations

For large tables with millions of files across thousands of partitions, this phase can be time-consuming and generates significant API costs. The listing happens in parallel across Spark worker nodes, with parallelism determined by the number of unique directory paths.

Phase 2: Delta Log Comparison

VACUUM reads the Delta transaction log to identify which files are currently referenced. Files are marked for deletion if they:

1. Are not referenced in the current table state, AND

2. Are older than the retention threshold (7 days by default)

Critically, VACUUM FULL doesn’t just consult the Delta log—it also identifies straggler files: files that exist in the table directory but were never successfully committed to the Delta log (typically from aborted writes or failed jobs).

Phase 3: Deletion

File deletion is a driver-only operation. The driver issues deletion commands to the cloud storage provider:

• AWS: Uses DeleteObjects bulk API (single-threaded)

• Azure/GCP: Can delete in parallel if spark.databricks.delta.vacuum.parallelDelete.enabled is set to true

The Performance Challenge

For large tables, VACUUM FULL can take a significant amount of time—reports from the community mention runs lasting 30-60 minutes or more for petabyte-scale tables. The recursive listing phase is often the bottleneck, both in runtime and API call costs.

Teams typically run VACUUM FULL weekly or monthly due to these performance constraints.

VACUUM USING INVENTORY: The Extreme Scale Solution (Delta Lake 3.2.0)

Released in May 2024 with Delta Lake 3.2.0, VACUUM USING INVENTORY introduced a novel approach to eliminate the expensive file listing phase.

The Core Innovation

Instead of making live API calls to list files, this mode leverages pre-generated inventory reports from cloud providers:

• AWS S3 Inventory

• Azure Storage Blob Inventory

• GCP Storage Insights

These services generate daily or weekly manifest files containing complete lists of all objects in a bucket. VACUUM USING INVENTORY:

1. Reads the pre-generated inventory manifest

2. Compares it against the Delta transaction log

3. Identifies and deletes files

The Reality: When NOT to Use It

Despite impressive performance numbers, VACUUM USING INVENTORY comes with significant operational overhead that makes it unsuitable for most organizations:

Setup Complexity:

• Configure cloud inventory service

• Establish inventory scheduling (daily/weekly)

• Create and maintain manifest tables in Delta Lake

• Configure VACUUM to read from inventory locations

• Monitor inventory freshness and handle schema evolution

The Compliance Risk:

Recommendation:

VACUUM USING INVENTORY is best reserved for organizations that:

• Operate dozens of petabyte-scale tables

• Have dedicated platform engineering teams

• Have low compliance risk or robust inventory monitoring

For most teams, the operational complexity and risk aren’t justified by the savings. Standard VACUUM modes are simpler, more reliable, and sufficient.

VACUUM LITE: The Practical Evolution (Delta Lake 3.3.0)

On January 6, 2025, Delta Lake 3.3.0 introduced VACUUM LITE, which fundamentally changed the maintenance equation by addressing VACUUM FULL’s core bottleneck in a simpler way.

The Key Insight

Rather than scanning storage directories or managing complex inventory systems, VACUUM LITE takes a direct approach:

1. Read the Delta transaction log (which already lists all committed files)

2. Identify files marked as “removed” and older than the retention threshold

3. Delete them

That’s it. No directory traversal. No ListObjects API calls. No inventory management.

VACUUM events LITE RETAIN 168 HOURS

Performance Characteristics

Community reports show dramatic speedups:

• Operations that took 30-60 minutes with VACUUM FULL now complete in under 5 minutes

• Minimal API call costs since directory listing is eliminated

• Suitable for frequent execution (daily or multiple times per day)

The Trade-Off: Speed vs. Thoroughness

VACUUM LITE’s speed comes from being more selective. It only deletes files tracked in the Delta transaction log. It will not catch:

• Straggler files from aborted writes

• Orphaned files from failed jobs

• Any files not committed to the transaction log

The Baseline Requirement

There’s a critical safety mechanism: VACUUM LITE requires at least one successful VACUUM FULL run within the transaction log retention window (30 days by default).

This baseline ensures VACUUM LITE has a complete picture. Without it, the command fails with:

DELTA_CANNOT_VACUUM_LITE: VACUUM <tableName> LITE cannot delete all eligible files 
as some files are not referenced by the Delta log. Please run VACUUM FULL.

This safety check prevents accidental data loss. If you encounter this error, run VACUUM FULL once to establish the baseline.

Recommended Strategy: The Hybrid Approach

The most effective strategy combines both VACUUM modes:

Weekly Maintenance: VACUUM LITE

Use LITE for routine cleanup of files from normal operations (MERGE, UPDATE, DELETE):

-- Scheduled daily
VACUUM high_churn_table LITE RETAIN 168 HOURS

When to use:

• Tables with frequent data modifications

• Well-managed ingestion pipelines

• Daily or multi-daily maintenance windows

Weekly/Monthly Deep Clean: VACUUM FULL

Use FULL for comprehensive cleanup:

-- Scheduled weekly (e.g., Sunday morning)
VACUUM high_churn_table RETAIN 168 HOURS

When to use:

• Establishing the baseline for LITE mode

• After irregular ingestion issues

• Cleaning up straggler files

• Compliance-critical scenarios requiring thoroughness

• Periodic deep maintenance

This two-tier approach provides:

• Fast, frequent cleanup via LITE (minutes daily)

• Comprehensive straggler removal via FULL (weekly/bi-weekly)

• Predictable maintenance windows

• Lower overall compute costs

The Secret to a Cheaper VACUUM (Hardware Configuration) $$$

The standard Databricks recommendation for VACUUM is:

• Auto-scaling cluster (1-4 workers)

• Compute-optimized instances (AWS C5, Azure F-series, GCP C2)

• 8 cores per worker

• 8-32 core driver

This is a safe, balanced configuration. However, there’s an alternative approach if you can afford some occasional failures but want to save every penny.

The Single-Node Strategy

For VACUUM LITE and certain VACUUM FULL scenarios, running on a single powerful driver node (0 workers) can be more cost-effective:

Configuration:

• Driver: Large compute-optimized instance (32-64 cores)

• Workers: 0

• Mode: Single-node cluster

Why this works:

1. File listing: For many tables, the listing phase is bottlenecked by cloud storage API rate limits, not CPU. Additional workers may not significantly improve listing time.

2. File deletion: This is driver-only regardless of worker count. Workers sit idle during this phase.

3. Cost: A single large driver for 30 minutes often costs less than a multi-node cluster for the same duration..

Decision Framework: Choosing Your VACUUM Strategy

Here’s a practical guide for determining which VACUUM mode to use:

Use VACUUM FULL When:

✅ Running your first VACUUM on a table (establishes LITE baseline)
✅ After messy or failed data ingestion jobs
✅ Periodic deep cleaning (weekly/bi-weekly/monthly)
✅ Compliance-critical scenarios (thoroughness matters)
✅ Tables with known straggler file issues
✅ Resolving DELTA_CANNOT_VACUUM_LITE errors

Typical frequency: Weekly to monthly, depending on table characteristics

Use VACUUM LITE When:

✅ Daily or frequent maintenance operations
✅ Well-managed tables with regular modifications
✅ Routine cleanup of committed files
✅ Tables with an established FULL baseline
✅ Speed and cost optimization are priorities

Typical frequency: Daily to weekly

Understanding the Timeline: When Features Became Available

For teams planning their VACUUM strategy, here’s when each feature was introduced:

• VACUUM (FULL mode): Original Delta Lake feature, available since early versions

• VACUUM USING INVENTORY: Delta Lake 3.2.0 (May 9, 2024)

  • Requires Databricks Runtime 15.4 LTS or later

• VACUUM LITE: Delta Lake 3.3.0 (January 6, 2025)

  • Requires Databricks Runtime 16.1 or later

Check your Databricks Runtime version to determine which features are available in your environment.

Key Takeaways

The evolution of VACUUM in Delta Lake reflects the maturation of large-scale data lakehouse operations:

1. VACUUM FULL remains essential for comprehensive cleanup and establishing baselines, despite being slower and more expensive

2. VACUUM LITE is the game-changer for most teams—fast enough for daily use, simple enough to trust in production, and cost-effective

3. VACUUM USING INVENTORY is specialized and best reserved for extreme-scale scenarios with dedicated platform engineering. Compliance considerations trump cost savings: For regulated data, thoroughness and reliability are more important than marginal compute savings

4. The hybrid approach works: Combine frequent LITE runs with periodic FULL runs for optimal cost and coverage

5. Hardware optimization matters: Test both standard auto-scaling and single-node configurations to find your optimal cost/performance balance

The lesson isn’t “always use the latest feature”—it’s understanding the trade-offs and choosing the right tool for your specific tables, scale, and requirements.

References

1. Delta Lake 3.3.0 Release Notes - VACUUM LITE introduction

2. Delta Lake 3.2.0 Release Notes - VACUUM USING INVENTORY

3. Databricks Documentation: Remove unused data files with VACUUM

4. Efficient Delta Vacuum with File Inventory

5. Delta Lake GitHub Repository

But what if we could just bring this process directly to Slack? This blog post will walk you through building a Slackbot that enables real-time agent interaction and feedback collection.

How does tracing and feedback work in MLflow?

With MLflow Production Monitoring, you can see traces arrive directly in an MLflow experiment. These traces can be synced to a table in Unity Catalog.

Each trace has a unique ID automatically generated by MLflow. This ID can be used to add feedback (source: Databricks documentation) via the MLflow log_feedback function. This can be an LLM judge or human feedback. Feedback is also stored as an assessment linked to the specific trace, making it queryable through MLflow.

A labeling session (source: Databricks documentation) is a special type of run within MLflow. Databricks recommends adding specific traces to a labelling session beforehand-- the custom or built-in review app then connects to that labeling session and exposes the traces to SMEs. The app allows us to just interact with the MLflow client in a specific way. This requires us to pre-select traces.

To create a Slackbot that can perform the same tasks as a custom review app, we will need to host it on a Databricks App. In this app, we are going to use labeling sessions slightly differently. Instead of interacting with pre-selected traces, we will allow SMEs to interact with the agent directly, creating traces and adding them to an already- created labeling session immediately. Then, the SME can add feedback via Slack interactions.

Building the Slackbot Review App

Follow along with the code here.

This is the experience we want:

1. Human experts ask questions in a Slack channel.

2. The agent answers the question in the same Slack thread.

3. Human experts provide feedback via Slack shortcuts.

Therefore, our Slackbot should:

1. Listen to messages in Slack.

2. Call our agent in Databricks.

3. Collect feedback from SMEs in Slack.

4. Annotate MLflow traces with that feedback.

   

Some housekeeping

Before we get started with writing the Databricks app, we will first need to create the following:

Creating an app in Slack

First, let’s create an application in Slack. For more detailed instructions, check out this Medium blog post.

I have included the app manifest for the Slackbot with all necessary configurations, but check out the necessary scope and permissions for the bot. We will definitely need the scopes: (1) chat:write (2) groups:read (3) im:read (4) mpim:history (5) commands.

Once you have installed the app in Slack, you will be given a Bot User oauth token. Save this securely. We will need to use that in our app.

Creating a Databricks App

Databricks Apps makes hosting straightforward, as each app has an associated Service Principal. All we need to do is ensure that the Service Principal has access to our MLflow experiment and agent endpoint.

Using the CLI, we can create the app:

databricks apps create slackbot

Sync local files to the Databricks workspace:

databricks sync . “/Users/$DATABRICKS_USERNAME/slackbot-app”

Then, deploy:

databricks apps deploy slackbot --source-code-path /Workspace/Users/$DATABRICKS_USERNAME/agent-proto

Creating a MLflow labeling session

We should also create a labeling session that we will use within our MLflow experiment. This creates a persistent Mlflow run that we will link all Slack-generated traces to. You can do this in a notebook with the SDK or through the MLflow experiment UI.

import mlflow.genai.labeling as labeling

import mlflow.genai.label_schemas as schemas

# Create a simple labeling session with built-in schemas

session = labeling.create_labeling_session(
    name=”customer_service_review_jan_2024”, 
    assigned_users=[”alice@company.com”, “bob@company.com”],
    label_schemas=[schemas.EXPECTED_FACTS]  
    # Required: at least one schema needed 
)

Source: Databricks documentation.

1. Initializing the Slack client

In our Databricks App, using the Slack SDK, we can easily connect to our Slack App:

def get_slack_auth():
    w = WorkspaceClient()
    token_bot = dbutils.secrets.get(scope=”brickbrain-scope”, key=”slack-bot-token”)    
    return token_bot

def start_slack_client():
    logger.info(”Initalized slack client. “)
    ssl_context = ssl.create_default_context()
    ssl_context.check_hostname = False
    ssl_context.verify_mode = ssl.CERT_NONE
    token_bot = get_slack_auth()
    client = slack_sdk.WebClient(token=token_bot, ssl=ssl_context)
    return App(client=client, process_before_response=False)

app = start_slack_client()

Note: store the Slack token in Databricks Secrets for security. Ensure your Service Principal has permissions to that secret scope.

2. Listening to events

Depending on the permissions given to your application, your slackbot will be able to receive and be able to respond to different events. Take a look at the full list of the events (source: Slack documentation).

First, let’s take a look at the message event, which observes whether or not a message was sent to a channel. In the example, I am observing every event that is sent to a channel. If you want to minimize the scope, you can select a message subtype or naively use string manipulation. I am going to be using slack-bolt moving forward to respond and take actions as the bot.

Bolt has many decorators that we can use to listen or observe events. For example, when observing the message event, I can declare the following:

@app.event(”message”)
def llm_response(event, say, client):
    logger.info(f”Message received - User: {event[’user’]}, Text: {event[’text’][:20]}...”)
   <...>

For different types of “listeners”, we can have different function arguments:

• payload: also accessible via the alias corresponding to the method name that the listener is passed to (message, event, action, shortcut, view, command, options).

  • In this case, event == payload

• say: function send a message to the channel associated with the event.

• ack: function that must be acknowledged that an incoming event was received by the app.

• client: web API client that uses the token associated with that event.

• logger

This is not a complete list! But these are the most important ones for our use case (source: Slack documentation).

3. Calling the agent

In our app, we want to respond to messages sent to the channel. We can easily trigger an LLM call now. However, in order to add feedback to the trace, we need to get the trace ID. When interacting with a Databricks endpoint, we can do this by setting the variable return_trace to True.

        input_data = {
            “input”: history + [{”role”: “user”, “content”:  message_text}],
            “databricks_options”: {”return_trace”: True}
        }

        response = mlflow_client.predict(endpoint=ENDPOINT_NAME, inputs=input_data)

The response output will then give me the trace ID:

        trace_id = response[’databricks_output’][’trace’][’info’][’trace_id’]

4. Responding to the message

To respond within a thread, we will need to use the client API. Recall that the listener argument “say” is offered with most events. However, “say” does not allow us to respond within a thread.

LLMs often use and produce Markdown as an output format. It is important to note that Slack uses its own markdown language, and although most basic syntax support is provided, some elements are absent. Take a look at what is supported here.

If you want to ensure that the output is stylized in the same way that the LLM intended, I would suggest looking at manually converting the Markdown text into Slack’s mrkdwn format. This would require some string manipulation with regex (source: Github repo).

    result = client.chat_postMessage(
        channel=event[’channel’],
        blocks=[
            {
                “type”: “section”,
                “text”: {”type”: “mrkdwn”, “text”: slack_response}
            },
        ],
        text=slack_response,
        thread_ts=event[’ts’],  # reply in the thread
        metadata={
            “event_type”: “agent_response”,
            “event_payload”: {
                “trace_id”: trace_id, # trace id in metadata
                “thread_id”: event[’ts’],
                “resource_type”: “AGENT_RESPONSE”,
            }
        }
    )

Using the Client API, we can also attach metadata to each message. This makes it easier to retrieve information across sessions, like trace_id.

We have designed the response simply, but Slack has a lot of options on how to design a Slack message. Take a look at Block Kit Builder to see how you can structure your Slack message with buttons, dividers, images, inputs, etc.

5. Adding feedback

We will use a Slack message shortcut to log feedback. I found this method to be the most straightforward and easiest to customize. However, we can also use Slack message blocks to design a feedback form as well.

When I use the add_feedback shortcut, this triggers the event “message_shortcut”. Because we have added the trace id to the metadata of the agent response Slack message, we can access that trace_id in the Slack shortcut.

@app.message_shortcut(”log_feedback”)
def handle_log_feedback_shortcut(ack, shortcut, client):
    ack()
    logger.info(f”Feedback message shortcut triggered by user: {shortcut[’user’][’name’]}”)
    
    message = shortcut[’message’]
    message_ts = message[’ts’]
    
    metadata = message.get(’metadata’, {})

When handling this event, we can use the Client API to open a view with the formatted feedback form. We can add comments and binary feedback. These inputs will be translated as input for mlflow.log_feedback(). However, log_feedback can take all sorts of values: integers, floats, categorical values, and multiple-category feedback (source: Databricks documentation). So, feel free to customize this to what your evaluation system needs.

Since this is a form, once we hit submit, we will need to respond to another Slack event as well. This will create another Slack event called “view”. This is where we actually handle the feedback submission and use mlflow.log_feedback(). For your review app, you can also log expectations (aka ground truth) using another function log_expectations().

6. Linking everything to a labeling session

We still have not linked these traces to a labeling session. To do so, we fetch the run ID associated with the labeling session and the trace_id:

def link_traces_to_run(run_id: str, trace_ids: List[str]) -> Dict[str, Any]:
    creds = get_databricks_host_creds()
    url = _get_mlflow_api_url(’/traces/link-to-run’, creds=creds)
    data = {’run_id’: run_id, ‘trace_ids’: trace_ids}

############################
in the @app.event function: 
############################

link_traces_to_run(run_id=LABELLING_SESSION.mlflow_run_id, trace_ids=[trace_id])
logger.info(f”Traces linked to run - Run: {LABELLING_SESSION.mlflow_run_id}, Trace: {trace_id}”)

7. Handling with conversation history

Slack threads make conversation history management simple. Instead of requiring a database to checkpoint, we can simply fetch the threads themselves. Using the client API and the thread ID:

def get_thread_messages(client, channel, thread_ts):
    response = client.conversations_replies(
        channel=channel,
        ts=thread_ts,
        inclusive=True,  # Include the parent message
        limit=10  # Max messages to retrieve
    )
    logger.info(f”Retrieved {len(response[’messages’])} messages from thread {thread_ts}”)
    return response[’messages’]

Happy reviewing!

Take a look at the full implementation and code here.

There are no limitations in how you can use MLflow review apps! You can easily bring the feedback mechanism in MLflow to Slack, reducing any friction in the feedback process. Thanks for reading.

Classification Machine Learning problems have been around for a long time. Traditional approaches like decision trees and SVMs have dominated the landscape, and with the rise of NLP, we gained powerful tools for text classification using techniques like TF-IDF and word embeddings. Now, we have Large Language Models that understand context and nuance in ways that traditional models often can’t match. This opens up opportunities to automate classification processes, especially in complex scenarios where rigid classification rules break down, such as hierarchical classification.

How can you take advantage of the latest frontier in classification? This blog will show you how to use AI_QUERY in Databricks to run batch inference for complex hierarchical classifications with LLMs.

As always, choose the best approach for your problem! There are many occasions when a classical approach is a better choice. However, if:

• Your historical data is not trustworthy (e.g. it may be inconsistent, difficult to validate, or labeled by multiple people with different interpretations; you can’t trust that the patterns held therein are going to be useful in a classical machine learning setup)

• Your categories are flexible (e.g. you need to be able to add a new category on the whims of your business partners without retraining an entire model)

• You’re dealing with nuanced, context-dependent classifications (e.g. rigid rules fail; maybe “billing dispute” vs “billing inquiry” depends on subtle linguistic cues that are difficult to encode as features)

… then read on to learn how to use AI_QUERY for hierarchical classification!

The Goal

Let’s say that we’re a telecommunications company and we have a lot of customer call transcripts coming in that we want to classify along four levels: Domain (Level 1) → Category (Level 2) → Problem Type (Level 3) → Root Cause (Level 4). This hierarchy creates a comprehensive support taxonomy covering everything from network outages and billing disputes to device issues and order management, so our business team can gain a lot of insights once all of these transcripts are classified correctly. The only obstacle is actually classifying these transcripts. How should we do that?

If you’re already familiar with the power of AI_QUERY, which allows you to query LLM endpoints via SQL for batch workloads, you could jump right into the Databricks SQL editor or a notebook and use the SQL function right away, relying on a monster prompt that attempts to take each record and assign the right hierarchy in one shot.

The Problem with the “Everything at Once” Approach

Unfortunately, if we have 12 Domains, and each Domain contains 5-6 Categories, and each Category has 3+ Problem Types, and each Problem Type has 5+ Root Causes, we have at minimum 900 hierarchical paths that a single transcript could take.

Combinatorial explosion! From a statistical perspective, we’re asking an LLM to perform a 900-class classification problem—the kind of task where even specialized neural networks start sweating.

There are a couple of issues with treating each of the 900 options as its own unique value:

1. More choices = worse outcomes: Imagine a 900x900 confusion matrix; a classification problem of this size will very likely result in poor precision and poor recall.

2. The “needle in a haystack” problem: When you present an LLM with a massive list of options, accuracy degrades significantly. False Negatives are easy when there are 899 other options, some of which sound very similar.

3. Prompt complexity becomes unmaintainable (and more expensive): Your prompt becomes a short story, making it difficult to debug, version control, and understand what instructions the model is actually following - and on top of that you’re paying for all those tokens each time you send a request!

What About Individual Level Classification?

On the other hand, if we look at each level individually, we greatly reduce the number of options per record. We only need to predict across 12 Domains, for example, and once we have the Domain, we only need to predict across 5 or 6 Categories, and so on down the levels. This approach will improve shrink the prediction space and likely improve accuracy, precision, and recall. The downside here is that we’re now running four sequential SQL queries. Depending on the model we’re using, this could get expensive and/or slow.

By now you might be thinking, “So that porridge is too cold, and this one is too hot… How do I use AI_QUERY for hierarchical classification???”

The Goldilocks Solution: Hierarchically-Aware Classification

To get just-right porridge, we can find a balance of the two approaches above by running two queries: the first will use a simple prompt to classify Level 1, and then the second will use a dynamic prompt to classify Levels 2-4. Here’s our recipe:

1. Create a hierarchies table

2. Build a gold-standard evaluation dataset

3. Execute your Domain (Level 1) prompt

4. Execute your Levels 2-4 prompt (with dynamic hierarchy filtering)

Step 1: Build Your Hierarchy Table

First, we need to build a table of our classification hierarchies. This table will be the reference for our second dynamic AI_QUERY prompt. It should look something like the schema here:

transcript_classification_hierarchy_table
├── level_1 (string)             -- Domain
├── level_2_map (string)         -- Category
├── level_3_map (string)         -- Problem Type
└── level_4_map (string)         -- Root Cause

Each row represents a valid path through your hierarchy. This is your source of truth that ensures we only provide valid paths to the LLM.

Here’s a sample record:

Step 2: Build Your Evaluation Dataset

Next, we need to make sure we have a solid evaluation dataset. If possible, grab an unsuspecting nearby SME and beg them to help you build this dataset.

The number of examples depends on how many categories you have in your dataset, but aim for at least 50-100 labeled examples covering your major categories. Try to avoid a scenario where you spend a lot of time prompt engineering, only to realize your ground truths are not all that accurate to begin with and you’ve been tuning your prompt in the wrong direction (speaking from personal experience, this is not fun).

Bonus: Having a gold-standard evaluation dataset allows you to continually try new models and model versions as they’re released and use the cheapest, fastest version that passes your evaluation metrics. This is deeply unsexy work, but it is possibly the most valuable.

Step 3: Classify Level 1 (Domain)

Once you have your hierarchical table and evaluation dataset, we’re ready to run our first AI_QUERY SQL statement: classifying Level 1. In this first batch inference round, we focus only on classifying the Domain. Since all the hierarchies stem from this first choice, getting it right is critical—but with only 12 options, our LLM has a much better chance of getting the classifications right. Here’s a sample query, using Llama 3.3 70b. We concatenate the prompt, which consists of the 12 Domain options, as well as the transcript itself and an instruction to only return the category name.

%sql
CREATE OR REPLACE TABLE catalog.schema.call_transcripts_l1_predictions AS
SELECT
   call_id,
   transcript_text,
   AI_QUERY(‘databricks-meta-llama-3-3-70b-instruct’,
      CONCAT(
       ‘Classify the transcript into one of the following categories:
       - Network & Connectivity
       - Billing & Payments
       - Account Management
       - Device & Equipment
       - Service Provisioning
       - Technical Support
       - Mobile Services
       - Internet Services
       - TV & Streaming
       - Voice Services
       - Security & Privacy
       - Sales & Orders\\n\\n’,
       ‘Transcript: ‘, transcript_text, ‘\\n\\n’,
       ‘Return only the category name.’     
    )  
  ) 
AS level_1_classification
FROM catalog.schema.call_transcripts_raw

Spend time here refining the prompt and testing out a variety of models to get high accuracy at Level 1 before moving on to the next step. In this example, my prompt is very simple: I don’t even bother defining my categories. In your own use cases, adding in descriptions of each category may be beneficial.

Step 4: Classify Levels 2-4

Once we’re happy with the Level 1 accuracy, we create our second AI_QUERY SQL statement. This time, we will write a much more complex prompt that 1) builds upon the L1 classifications and 2) joins with the hierarchical mappings from Step 1. This way, our prompt only includes the valid options for Levels 2 through 4 that correspond to the predicted Level 1 Domain.

%sql
CREATE OR REPLACE TABLE catalog.schema.call_transcripts_all_classifications AS
SELECT 
  call_id,
  transcript_text,
  l1.level_1_classification,
  AI_QUERY(‘databricks-meta-llama-3-3-70b-instruct’,
    CONCAT(
      ‘Classify the call transcript into Level 2, Level 3, and Level 4 subcategories. Strictly adhere to the hierarchies as listed below.\n\n’,
      ‘Transcript: ‘, l1.transcript_text, ‘\n’,
      ‘Level 1: ‘, l1.level_1_classification, ‘\n\n’,
      ‘Valid Level 2 options: ‘, hier.level_2_map, ‘\n\n’,
      ‘Valid Level 3 options by Level 2: ‘, hier.level_3_map, ‘\n\n’,
      ‘Valid Level 4 options by Level 3: ‘, hier.level_4_map, ‘\n\n’,
      ‘Return ONLY valid JSON (no markdown): {”level_2”: “X”, “level_3”: “Y”, “level_4”: “Z”}\n’
    )
  ) AS classification_json
FROM catalog.schema.call_transcripts_l1_predictions l1
INNER JOIN catalog.schema.transcript_classification_hierarchy_table hier
   ON l1.level_1_classification = hier.level_1

The query above might feel a bit abstract, so let’s take a look at what the prompt will actually look like for the L1 Domain “Billing & Payments”:

Classify the call transcript into Level 2, Level 3, and Level 4 subcategories. Strictly adhere to the hierarchies as listed below.

Transcript: [transcript not shown for brevity]

Level 1: Billing & Payments

Valid Level 2 options: Auto-Pay, Invoice Issues, Payment Plans, Payment Processing, Refunds

Valid Level 3 options by Level 2: {”Invoice Issues”: [”Cannot Access Invoice”, “Incorrect Charges”, “Missing Credits”], “Payment Processing”: [”Duplicate Payment”, “Payment Declined”, “Payment Not Posted”], “Auto-Pay”: [”Cannot Disable”, “Not Working”, “Setup Failed”], “Payment Plans”: [”Application Denied”, “Early Payoff Request”, “Missed Payment”], “Refunds”: [”Incorrect Amount”, “Not Received”, “Request Denied”]}

Valid Level 4 options by Level 3: {”Incorrect Charges”: [”Double Billing”, “Proration Error”, “Service Not Ordered”, “Tax Calculation Error”, “Wrong Rate Applied”], “Missing Credits”: [”Adjustment Not Posted”, “Manual Credit Not Entered”, “Promotion Not Applied”, “Refund Not Processed”, “System Processing Delay”], “Cannot Access Invoice”: [”Account Access Restricted”, “Email Not Received”, “PDF Generation Error”, “Portal Login Issue”, “System Maintenance”], “Payment Declined”: [”Bank Fraud Detection”, “Card Expired”, “Incorrect Card Information”, “Insufficient Funds”, “Payment Gateway Error”], “Payment Not Posted”: [”Bank Processing Time”, “Manual Entry Required”, “Processing Delay”, “System Synchronization Issue”, “Wrong Account Number Entered”], “Duplicate Payment”: [”Auto-Pay and Manual Payment”, “Browser Refresh Error”, “Customer Submitted Twice”, “Multiple Payment Methods Active”, “System Glitch”], “Not Working”: [”Auto-Pay Disabled”, “Bank Account Closed”, “Insufficient Funds”, “Payment Method Expired”, “System Configuration Error”], “Setup Failed”: [”Account Not Eligible”, “Bank Verification Failed”, “Incompatible Payment Type”, “Invalid Account Information”, “Portal Technical Issue”], “Cannot Disable”: [”Balance Owed”, “Contractual Requirement”, “Pending Transaction”, “Portal Access Issue”, “System Processing Lag”], “Application Denied”: [”Account Not Eligible”, “Balance Too Low”, “Credit Check Failed”, “Existing Plan Active”, “Previous Default History”], “Missed Payment”: [”Auto-Pay Not Set Up”, “Customer Forgot”, “Financial Hardship”, “Payment Date Confusion”, “Payment Method Failed”], “Early Payoff Request”: [”Approved With Fee”, “Approved Without Fee”, “Denied Per Terms”, “Requires Balance Verification”, “Under Review”], “Not Received”: [”Check Lost in Mail”, “Processing Time Not Elapsed”, “Refund Method Changed”, “System Error”, “Wrong Bank Account”], “Incorrect Amount”: [”Calculation Error”, “Credits Applied First”, “Fees Deducted”, “Partial Refund Per Policy”, “Tax Adjustment”], “Request Denied”: [”Contractual Terms”, “Outside Refund Window”, “Previous Refund Given”, “Promotional Restriction”, “Service Already Used”]}

Return ONLY valid JSON (no markdown): {”level_2”: “X”, “level_3”: “Y”, “level_4”: “Z”}

Now, instead of asking the LLM to consider 900 paths, we’re saying, “Hey, you already decided this is a ‘Network & Connectivity’ issue. Here are the valid paths within that domain. Pick one.” We’ve gone from a 900-class problem to a ~75-class problem. Our accuracy, precision, and recall will benefit.

Now We Have Our Hierarchical Classifications!

With this approach, you get:

• Improved accuracy by reducing overall cognitive load on the LLM

• Lower token costs by only including relevant options in each prompt

• Valid hierarchies that make sense together

• Easier debugging and iteration when you evaluate Level 1 accuracy independently

• Flexibility to add new categories without rewriting your entire classification logic

Wrapping Up

AI_QUERY in Databricks makes it remarkably easy to leverage LLMs for complex classification tasks without managing API calls, rate limits, or infrastructure. By combining it with a hierarchically-aware approach, you can tackle complex classification problems. Break down the complexity, give your LLM manageable choices at each step, and let the structure of your domain guide the way.