Every company depends on a handful of KPIs: revenue, active users, churn, margin, and each one usually gets defined in a dozen places at once: a SQL query here, a spreadsheet formula there, a dashboard filter somewhere else, each subtly different, until two teams walk into a meeting with two revenue figures and no one can say which is right. A semantic layer fixes that by providing a single governing definition that every tool reads, and Unity Catalog Metric Views are Databricks’ version of it.

That consistency has always mattered, but AI raises the stakes. When an agent answers a business question at machine speed, a wrong or ambiguous definition no longer just misleads one analyst; it gets baked into automated decisions and customer-facing answers. The semantic layer is where trust in your data is enforced, and your agents are only as good as the definitions behind them. This post is about the newest consumer of that layer, the AI agent, and a lightweight pattern that lets it reach the semantic layer using only existing Databricks services.

Agents can’t find the right metric definition

A person finds the right metric view without thinking about it. An agent starts from nothing. When a custom agent receives a question like “what was our average order value this quarter?”, it first has to find the right metric view from among hundreds across dozens of domains, and then understand its measures, dimensions, and query syntax.

Today, there’s no first-class, public way to search across them. An agent can already reach the definitions: it can enumerate catalog objects and read a metric view’s definition through the Unity Catalog REST API or SQL, and run one with the MEASURE() function. The trouble is choosing. Enumerating and parsing raw definitions on every call doesn’t scale past a small catalog. It re-scans each time, burns tokens, and surfaces overlapping or conflicting definitions with no way to rank them.

The semantic layer is a valuable, governed asset, but there’s no discovery surface that lets an agent ask, “Given this question, which metric view should I use, and what does it expose?” the way it would for any other object it can reach. This post builds that surface.

Expose the semantic layer as MCP

The idea is a small, customer-owned Databricks App that acts as an MCP server with a single job: answer “which metric views match this ask, and how do I query them?”

At startup, the app builds an in-memory index of trimmed metric-view metadata: each view’s name, description, measures, and dimensions. It exposes one MCP tool, search_metric_views. An agent calls it with a natural-language ask and gets back the matching metric views, their measures and dimensions, and an example query so the agent knows how to use them without having to learn the metric-view syntax on its own.

Two design choices matter:

• It’s discovery, not execution. The app returns which metric views fit and an example of how to query them.

• You control what’s exposed. The app can index only certified metric views, so external agents see governed, production-ready definitions and nothing that’s still in development or QA.

Architecture

Multiple agents, inside or outside Databricks, connect to one Databricks App, which acts as the MCP server and looks up the semantic layer in Unity Catalog. The pattern improves discoverability: the app returns matching metric definitions and an example query. The agent can take necessary actions to generate SQL or manage the semantic layer to avoid duplication, etc., based on the use case.

Where does the metadata live?

The index can live in a few places, and each earns its place in a different situation:

• In-memory, inside the app: the simplest option, and the one this post uses. No extra infrastructure, and reads are fast because they never leave the process. The catch is that the index rebuilds on every restart; each running copy of the app maintains its own copy; definitions can go stale between refreshes; and search is at the keyword level. Enough for a bounded catalog and a single team.

• Metadata table via a SQL endpoint: fits a bounded catalog that changes rarely. Every app instance reads one shared, governed copy, refreshed by a scheduled job.

• Lakebase: fits when many app instances and agents hit the index at once, and you want durable, low-latency lookups with structured filters such as domain, owner, or certified status, with room to keep usage or audit state alongside it.

• Managed AI Search (vector) index: fits when search quality is the point, whether a large catalog or plain-language questions, where ranking by meaning beats keyword matching, at the cost of more infrastructure and an index to keep in sync.

The right choice comes down to the size of the catalog, the number of requests per minute you expect, and the amount of latency you can tolerate.

Governance

The app runs under a service principal, and you govern what it can reach with Unity Catalog grants (for example, restricting it to certified metric views). This fits multi-agent and cross-environment scenarios where the calling agents don’t carry a Databricks user identity.

If you need user-level access control, on-behalf-of (OBO) user authorization is supported. The agent acts as the calling user, and row and column governance is preserved end-to-end. It adds identity-passing complexity and isn’t required for the service-principal pattern this post uses.

Prerequisites

Before you build the app, get a few things in place:

1. Databricks workspace

• Active Databricks workspace

• One or more Unity Catalog metric views

2. An MCP-capable agent or client: anything that speaks the Model Context Protocol can connect and call the search tool.

Setup Instructions

Step 1: Get the code

Download the five files from the repo folder:

https://github.com/adgitdemo/ad_databricks/tree/main/mcp-metric-view-search

You’ll have app.py, app.yaml, config.yaml, requirements.txt, and README.md.

Step 2: Point it at your warehouse and your data

Two files need real values before you deploy.

In app.yaml, set your SQL Warehouse ID. You’ll find it under SQL Warehouses > your warehouse > Connection details.

env:

- name: DATABRICKS_WAREHOUSE_ID

value: “your_warehouse_id”

In config.yaml, specify which catalogs and schemas the app should scan for metric views. Scope it down to keep the index focused, or leave it empty to scan everything the app can see.

scope:

- catalog: your_catalog

schemas:

- your_schema

Step 3: Deploy the app

Upload the folder to your workspace and deploy it. Deploying is what creates the app’s service principal, and that service principal is the identity you’ll grant data access to in the next step. So deploy first.

databricks workspace import-dir ./mcp-metric-view-search \

/Workspace/Users/<email id>/mcp-metric-view-search --overwrite

databricks apps create metric-view-search-app

databricks apps deploy metric-view-search-app \

--source-code-path /Workspace/Users/<email id>/mcp-metric-view-search

You can do the same thing from the Apps UI if you prefer.

Step 4: Find the app’s service principal

databricks apps get metric-view-search-app

Look at the service_principal_name field. It’ll be something like xxxx-mcp-metric-view-search. The app runs as this identity, so this is what needs to read your data, not your own user account.

Step 5: Grant the service principal access to your data

The app reads metric views through the warehouse, so the service principal needs three Unity Catalog grants plus warehouse access. Run each GRANT on its own. Databricks won’t accept several of them in a single statement.

GRANT USE CATALOG ON CATALOG your_catalog TO `xxxx-mcp-metric-view-search`;

GRANT USE SCHEMA ON SCHEMA your_catalog.your_schema TO `xxxx-mcp-metric-view-search`;

GRANT SELECT ON SCHEMA your_catalog.your_schema TO `xxxx-mcp-metric-view-search`;

Then give it the warehouse:

GRANT CAN USE ON WAREHOUSE `your_warehouse_id` TO `xxxx-mcp-metric-view-search`;

Repeat the schema grants for every schema you listed in config.yaml.

Step 6: Confirm it’s actually indexing

Redeploy so the app re-scans with its new permissions, then check any one of these:

- Open the app URL in a browser. The landing page shows how many metric views it has indexed.

- Hit the /health endpoint. It returns {”status”: “ok”, “indexed_metric_views”: N}.

- Read the app logs. You want to see the Indexed N metric view(s). If you instead see Cannot list catalogs or the scope still shows your_catalog_name, the config or the grants didn’t land.

Step 7: Let your teammates and their agents in

Everything above enables the app to read your data. To let other people connect their agents to it, grant them permission on the app itself. There are two ways to do this.

From the UI: open the app’s page, click Permissions in the top right, and add users, groups, or service principals. CAN USE is enough to connect and call the tools. CAN MANAGE also lets them redeploy and change settings.

Without the UI: use the permissions API. A PATCH adds people without disturbing anyone who already has access (a PUT would replace the entire list, so use PATCH here).

databricks api patch /api/2.0/permissions/apps/metric-view-search-app \

--json ‘{

“access_control_list”: [

{”user_name”: “teammate@yourcompany.com”, “permission_level”: “CAN_USE”}

]

}’

You can grant a group or another service principal the same way, and add several entries at once:

databricks api patch /api/2.0/permissions/apps/metric-view-search-app \

--json ‘{

“access_control_list”: [

{”user_name”: “teammate@yourcompany.com”, “permission_level”: “CAN_MANAGE”},

{”group_name”: “data-team”, “permission_level”: “CAN_USE”},

{”service_principal_name”: “<client-id>”, “permission_level”: “CAN_USE”}

]

}’

To confirm who has access, read the permissions back:

databricks api get /api/2.0/permissions/apps/metric-view-search-app

This is a different permission from Step 5. Step 5 controls what data the app can read. Step 7 controls who can use the app.

Step 8: Connect an agent

Register the app’s /mcp endpoint as an MCP server in your agent, give the connection a name like metric-view-search, and authenticate with a Databricks token.

For a connection that won’t expire after an hour, use a non-expiring Personal Access Token instead of a short-lived OAuth token. Then:

In the Claude Code CLI, that’s one command:

claude mcp add metric-view-search https://<your-app-url>/mcp \

--header “Authorization: Bearer <databricks-pat>”

In a coded agent using the Claude Agent SDK, you configure the same three things (name, URL, and the auth header) in the mcpServers option, and add the tools to allowedTools so the agent can call them. See steps here for wiring the app using the Claude Agent SDK:

options = ClaudeAgentOptions(

mcp_servers={

“metric-view-search”: {

“type”: “http”,

“url”: “https://<your-app-url>/mcp”,

“headers”: {”Authorization”: “Bearer <databricks-pat>”},

}

},

allowed_tools=[”mcp__metric-view-search__*”],

)

Either way, metric-view-search is just the connection name, and <databricks-pat> is the token. Once connected, the agent can search your metric views, read their definitions, and generate valid SQL. Other agents follow the same pattern.

▎ Note: a PAT is a standing personal credential that acts on your behalf until revoked. Fine for a demo; for shared or production use, prefer a service principal.

Other agents follow the same shape; they just differ in where you put the URL and the header. These steps may change based on your agent/tools.

Demo

With a few metric views registered, an agent connected to the app can explore the semantic layer directly:

• Ask “what metric views do we have around revenue?” and the app returns the matching views with their measures and dimensions. Where definitions overlap (e.g., a fee-only revenue view and a blended-revenue view), both are surfaced with their descriptions so the agent or user can tell them apart.

• For each match, the app returns an example query showing how to use the metric view.

When do I need it?

A few situations where searching the semantic layer directly is useful:

• Use a semantic layer from your own agents. You’ve standardized KPIs as metric views and want custom agents to use them without having to stand up additional consumer surfaces.

• Partners and BI tools built on Databricks that want to explore available metrics programmatically to build more intelligent experiences.

• Governing a large semantic layer. Data stewards find overlapping or duplicate metric views by keyword and owner across thousands of definitions, then audit and consolidate them.

• Standardizing across a multicloud, multi-agent estate. Metric views are being open-sourced, so the same pattern can front a semantic layer wherever it runs.

• Choose the right entry point: use notebooks with the @distributed decorator for rapid prototyping, and switch to the AIR CLI with YAML-based configs for long-running, multi-node production training.

• Pick the right accelerator for the job: attach a low-cost A10 or single H100 for orchestration, and reserve multi-GPU or multi-node H100 configurations for actual training workloads.

• Use UCVolumeDataset and serverless_gpu.data.DataLoader for efficient data loading with local caching, automatic distributed partitioning, and optimized prefetching keep GPUs fed without Spark overhead.

• Enable MLflow autologging and monitor the System Metrics tab for GPU utilization and memory; set a custom run name and MLFLOW_RUN_ID to resume previous training runs.

This post covers the practical lessons we’ve learned from working with teams in Databricks and with customers shipping real GPU training workloads. Think of it as the “things we wish someone had told us on day one” guide.

If you’re training or fine-tuning deep learning models on Databricks, AI Runtime is how you get access to GPUs without the usual infrastructure headaches. But like any powerful tool, knowing how to use it well makes all the difference.

What is AI Runtime? How Can It Help?

AI Runtime is a serverless model training product at Databricks built specifically for deep learning workloads. It brings GPU support to Databricks Serverless, so you can train and fine-tune models using your favorite frameworks (PyTorch, DeepSpeed, vLLM, Ray, etc.) without managing any infrastructure.

Before AI Runtime, running GPU workloads meant configuring clusters, picking the right drivers, managing autoscaling, and often negotiating GPU capacity directly with your cloud provider (AWS, Azure, or GCP). AI Runtime takes all of that off your plate.

Key benefits:

• Fully managed GPU infrastructure: You request the GPUs you need and start working. Pay only for what you use. On-demand by nature.

• Seamless scaling. Go from a single GPU experiment to multi-node distributed training without reconfiguring anything.

• A runtime dedicated for deep learning: Choose either a minimal base environment for full control over your dependencies, or a full-featured AI environment pre-loaded with popular ML frameworks. Either way, CUDA, NCCL, and drivers are already set up. For multi-node H100s, RDMA networking is configured out of the box.

• Natively integrated across notebooks, jobs, Unity Catalog, and MLflow. Your development workflow, data access, experiment tracking, and governance all work together.

Here are the 5 AI Runtime best practices we will cover in this Part 1 blog. Feel free to jump directly to the section that’s interesting for you:

• What are my hardware accelerator options?

• What are environments in AI Runtime?

• What are the 3 ways of using AI Runtime- notebooks, CLI and IDE?

• How can I use MLflow for tracing and observability?

• What are the different options for data loading for training?

1. Hardware Accelerator Options

The following accelerators are currently supported. All accelerators provision a single node.

• 1xA10- 24 GB-Small to medium ML and deep learning tasks

• 1xH100- 80 GB- Workloads that need more GPU memory than 1xA10 provides but do not require multi-GPU distributed training.

• 8xH100- 80 GB per GPU- Large-scale AI workloads that need distributed training or multiple GPUs

For multi-GPU workloads, choose a single-node 8xH100 instance (one node with 8 H100 GPUs). Partial multi-GPU configurations like 2xH100 or 4xH100 are not supported though.

For multi-node workloads, scale out using multiple 8xH100 nodes; for example, 4 nodes with 8 H100 GPUs each give you a total of 32 H100 GPUs. Or you can use multiple nodes of 1xA10.

Besides the options listed above, be sure to consult the latest documentation, as additional accelerator types may become available over time.

2. Environments

AI Runtime offers two managed Python environments, the default base environment and the Databricks AI environment.

Default base environment

• Minimal, includes only required packages: torch, cuda, and torchvision

• You want full control over your dependency stack and prefer to install only what you need

Databricks AI environment

• Pre-loaded with popular ML frameworks (PyTorch, Transformers, and more)

• You want a complete environment for training, fine-tuning, and experimentation without manual dependency management

You can also use a workspace base environment that a workspace admin has built for serverless GPU compute.

Environment Caching: Environments are cached across sessions to speed up startup times. When you reconnect to AI Runtime with the same environment configuration, previously installed packages may be available from cache, reducing setup time. However, this behavior is not guaranteed, so always ensure your notebook includes the necessary %pip install commands for reproducibility..

Import custom modules: Import custom modules by placing them under /Workspace/shared and adding the path to sys.path. Or You can also upload module files as Workspace files and import them directly.

import sys

sys.path.append(”/Workspace/Shared/my-project/src”)

from my_module import my_function

Some differences as compared to Classic Compute:

• Spark: AI Runtime is a Python-only environment; Spark is not available as a local runtime. So you cannot use PySpark functions but you can use Spark Connect for data loading.

• MLR: Pre-installed AI Runtime packages do not fully replace Databricks Runtime ML, because some ML libraries available in Databricks Runtime ML are not included.

3. Main Ways to Get Started: Notebooks, CLI and IDE

You have two primary ways for running workloads on AI Runtime: Notebooks and AIR CLI. Each has its strengths, and picking the right one for your use case makes the journey easier.

3a. Notebooks

Notebooks on AI Runtime let you develop interactively, iterate quickly, and visualize results in place. Simply choose your preferred accelerator (A10, 1×H100, or 8×H100), attach it to your notebook, and your code will execute directly on that GPU-backed node.

Multi-GPU (Single Node)

To run jobs across multiple GPUs on a single node, use the @distributed decorator. Wrap your training workflow inside a function and apply the decorator to it. This function becomes the entry point for distributed execution, so all the components - model initialization, data loading, and training logic - should be defined within it.
The @distributed API integrates with major distributed training libraries: PyTorch, DDP, FSDP and DeepSpeed

@distributed(gpus=8, gpu_type=’H100’)

def run_train(num_epochs, batch_size):

    # training code here


run_train.distributed(num_epochs=3, batch_size=1)

Multi-Node

Notebooks also support remote execution which lets you submit work to a different set of remote GPUs than your attached compute. To enable this, set remote=True in the @distributed decorator. This makes it easy to scale beyond a single node without interrupting your workflow.

@distributed(gpus=32, gpu_type=’H100’, remote=True)

def run_train(num_epochs, batch_size):

    # training code here

Ray

For a single node (A10 or 8xH100), you can just use ray.init().

For multi-node, use @ray_launch decorator

from serverless_gpu.ray import ray_launch


@ray_launch(gpus=16, gpu_type=’h100’)

def run_ppo(config) -> None:

	# your ray training code here


run_ppo.distributed(DictConfig(config))

Best practices:

• Pin your library versions: AI Runtime has regular releases. If you need reproducibility, pin the versions that work for your code.

• For distributed training, any environment variable needed by your training code should be set within @distributed. Note: Some environment variables must be set before importing a particular library so that it initializes with the correct configuration and hardware settings. This is specific to the library, not AI runtime.

• For remote multi-GPU H100 jobs, it’s more efficient to attach your notebook to a lower-cost accelerator such as an A10 or a single H100. Since the actual training runs on remote GPU nodes, your local notebook compute is only used for orchestration.

• Keep experimentation in notebooks, production in jobs: Notebooks are great for prototyping and debugging, but once your code is stable, you can move it to a scheduled job. For scheduling notebook jobs, use the Jobs API or Declarative Automation Bundles (fka Databricks Asset Bundles)

• For operations that do not require GPUs (for example, converting data formats, or exploratory data analysis), attach your notebook to a CPU cluster to preserve GPU resources.

• Note: Connection to your interactive compute auto-terminates after 60 minutes of inactivity.

3b. AI Runtime CLI

air CLI is a command-line tool for submitting and managing training jobs on Databricks Serverless GPU compute directly from your terminal, instead of using notebooks.The CLI uses YAML-based job configuration, and supports workspace-based and git-based code workflows. It’s aimed for users who prefer repos, IDEs, git, and CI/CD-style workflows over notebooks.

SGCLI follows a simple submit → schedule → execute → observe loop:

1. Define your workload in a train.yaml file (compute shape, environment, launch command).

2. Submit via sgcli run -f workload.yaml

3. AIR automatically provisions the requested GPU nodes, installs dependencies from requirements.yaml, and injects coordination environment variables

4. Your training code runs (e.g., via torchrun) across the provisioned nodes.

5. Observe progress through sgcli get logs, sgcli get status, or the MLflow UI for GPU metrics.

Here is a simple train.yaml

experiment_name: simple-training

environment:

  dependencies: requirements.yaml

env_variables:

  NCCL_DEBUG: “INFO”

secrets:

  HF_TOKEN: ‘my_scope/hf_token’

compute:

  num_accelerators: 8

  accelerator_type: GPU_8xH100

code_source:

  type: snapshot

    snapshot:

      root_path: /home/username/repo

      git:

        branch: main

command: torchrun --nproc_per_node=8 train.py

max_retries: 2

timeout_minutes: 90

usage_policy_id: abcd123-25b8-3e87-9a2c-f86eb19d101c

Here is the doc for AI CLI for more information.

Best practices for AIR CLI:

• Use CLI for multi-node, long-duration jobs: If your training run takes hours or days and spans multiple nodes, AIR CLI is the right tool. It handles the orchestration cleanly.

• Treat the workload.yaml as the source of truth. Keep the launch definition in version-controlled YAML rather than relying on notebook/UI state, so runs are easier to reproduce, hand off, and compare later.

• Check workload.yaml and requirements.yaml into your repo along with train.py for version tracking and reproducibility.

Custom Docker Images: You can run your own Docker images with AI Runtime using a simple CLI and YAML-based configuration. This is useful for workloads that need specific libraries, complex dependencies, fully reproducible environments, or standard images defined by your organization.

When to use AIR CLI vs. Notebooks:

Use notebooks (with the @distributed decorator) for rapid iteration and prototyping.

Use the AI Runtime CLI when you need to:

• Submit GPU training jobs directly from your terminal.

• Define jobs in YAML for version control and reproducibility.

• Run long (multi-day) or distributed (multi-node) training that shouldn’t depend on an interactive session.

3c. Connecting from the IDE terminal like Visual Studio or Cursor

You can connect to AI Runtime on serverless GPU compute directly from a terminal in your IDE through an SSH tunnel.

To connect from a terminal within your IDE, run the databricks ssh connect command with the --accelerator option.

databricks ssh connect --accelerator=GPU_1xA10

To connect and start the session in Visual Studio Code or Cursor, use the --ide option. The CLI opens an IDE window pointing to the home workspace folder.

databricks ssh connect --ide=vscode

4. Use MLflow for Tracking and Observability

On AI Runtime, MLflow is the main place to track and inspect training runs. AIR is natively integrated with MLflow for experiment tracking, model logging, metric visualization, and observability.

It’s recommended to use the latest version of mlflow (minimum 3.7 version)

For each run of your experiment:

• The System Metrics tab provides an overview of GPU, CPU, and disk performance, including key metrics such as utilization and memory usage. These are generated automatically for each run. Use these metrics to proactively identify bottlenecks, tune workloads (for example by adjusting batch size or data loading), and ensure resources are right-sized so you maintain high utilization without over-provisioning

• The Model Metrics tab surfaces key performance indicators logged during training, such as accuracy, loss, and other task-specific scores. Use these metrics to validate and compare experiments, ensuring that changes to model architecture, training configuration (for example, batch size, learning rate, and data preprocessing),etc actually improve model quality over time/

• Logs for processes running on each GPU are stored as artifacts and can be accessed from the MLflow UI. For notebook-based runs, the same logs are also visible directly in the notebook cell outputs. Use these logs to monitor training progress, debug training issues, and trace performance regressions across runs so you can quickly identify and fix problems in your training pipeline.

System Metrics

Logs are also available as artifacts in the MLflow run

Best practices:

• Autologging: MLflow supports automatic logging for popular frameworks like PyTorch, TensorFlow etc. You can enable it for your training job by adding just one of code, mlflow.autolog(). It automatically captures:

  • Metrics: standard training and evaluation measures (such as accuracy and F1 score);

  • Parameters: hyper-parameters, such as learning rate and number of estimators

  • Artifacts: output files, such as your trained model weights.

• Manual Logging: you can manually log any parameter, metric or artifact in your training code.

• The Serverless GPU API automatically launches an MLflow experiment with default name /Users/{WORKSPACE_USER}/{get_notebook_name()}. To overwrite it, set

import os

os.environ[”MLFLOW_EXPERIMENT_NAME”] = “/Users/<username>/my-experiment”

• It’s recommended to customize the MLflow run name. It enables you to restart from a previous run.You can customize the run name using the run_name parameter in mlflow.start_run(run_name=”your-custom-name) or in third-party libraries that support MLflow (for example, Hugging Face Transformers). Otherwise, the default run name is jobTaskRun-xxxxx.

• Resume previous training by setting the MLFLOW_RUN_ID from the earlier run: mlflow.start_run(run_id=”<previous-run-id>”)

• Set the `step` parameter in MLFlowLogger to reasonable batch numbers. MLflow has a limit of 10 million metric steps, so logging every single batch on large training runs can hit this limit

Monitor GPU resources:

If you are using Notebook, you can use GPU resources in the right side pane to monitor GPU health and utilization while your code runs on AI Runtime. The pane supports both single-node and multi-node workloads. The pane displays the following metrics for each GPU: GPU utilization percentage, GPU memory usage, Temperature.

GPU metrics

If you are using CLI, you can utilize System Metrics to view GPU metrics.

5. Data Loading

Tabular Data:

• Small Dataset: For small Delta tables, use Spark Connect to load the data. Then convert the Spark DataFrame to a pandas DataFrame with toPandas(). If needed, you can further convert it to a NumPy array using to_numpy(). This works well for Single Node

# Load Delta table

df = spark.read.format(”delta”).table(”your_catalog.your_schema.your_table”)

# Convert Spark DataFrame to pandas

pdf = df.toPandas()

# Optional: convert pandas DataFrame to NumPy array

arr = pdf.to_numpy()

• Large dataset: For large Delta tables that are too big to convert with toPandas(), export the data to a Unity Catalog volume and load it directly using PyTorch or Hugging Face. This approach avoids Spark overhead during training and works well for both single-GPU and distributed training workflows.

# Step 1: Export the Delta table to Parquet files in a UC volume

output_path = “/Volumes/catalog/schema/my_volume/training_data”

spark.table(”catalog.schema.my_table”).write.mode(”overwrite”).parquet(output_path)

# Step 2: Load the exported data directly using Hugging Face datasets

from datasets import load_dataset

dataset = load_dataset(”parquet”, data_files=”/Volumes/catalog/schema/my_volume/training_data/*.parquet”)

Unstructured Data:

For unstructured data such as images, audio, and text files stored in Unity Catalog volumes, our recommendation is to use UCVolumeDataset from the serverless_gpu.data package.

UCVolumeDataset is a PyTorch IterableDataset that copies each file from the volume to a fast local cache on first access and yields the cached local file path. Besides local caching it supports automatic partitioning of files and divided across DataLoader workers, so each (rank, worker) pair receives a non-overlapping slice.

For optimal performance, pair UCVolumeDataset with serverless_gpu.data.DataLoader, rather than the stock PyTorch DataLoader, as it is tuned for serverless GPU I/O and fetches and caches files concurrently while the GPU computes.

from serverless_gpu.data import UCVolumeDataset

from torch.utils.data import IterableDataset

from PIL import Image

import torchvision.transforms.functional as TF

class ImageDataset(IterableDataset):

    “”“Decodes each cached file path from UCVolumeDataset into a tensor.”“”

    def __init__(self, path_dataset: UCVolumeDataset):

        self._path_dataset = path_dataset

    def __iter__(self):

        for local_path in self._path_dataset:

            image = Image.open(local_path).convert(”RGB”)

            yield TF.to_tensor(image)

path_dataset = UCVolumeDataset(”/Volumes/catalog/schema/my_volume/images”)

dataset = ImageDataset(path_dataset)

For more info about data loading, here is the doc.

Best Practices:

• Load data inside the @distributed decorator: For distributed training, move data loading code inside the @distributed decorator. The dataset size can exceed the maximum size allowed by pickle, so it is recommended to generate the dataset inside the decorator.

from serverless_gpu import distributed

# This may cause a pickle error if the dataset is too large

dataset = get_dataset(file_path)

@distributed(gpus=8, gpu_type=’H100’)

def run_train():

   # Load data inside the decorator to avoid pickle serialization issues

   dataset = get_dataset(file_path)

   ...

When you build a UCVolumeDataset in the decorator, it uses torch.distributed rank info at runtime to automatically split files across ranks, so you don’t need a DistributedSampler for file-based volume data.

• Use UCVolumeDataset for multi-epoch training so files are cached locally on first access and reused, instead of copying the entire volume upfront with shutil.copytree when you may only read part of it.

• Cache data locally for multi-epoch training. For large datasets, copy datasets to /tmp for faster access across epochs:

import shutil
shutil.copytree(”/Volumes/catalog/schema/volume/dataset”, “/tmp/dataset”)

• Parallelize data fetching: Use serverless_gpu.data.DataLoader to parallelize fetching. It’s a torch DataLoader subclass optimized for serverless GPUs, with higher defaults for worker count and prefetching so files are loaded and cached while the GPU runs. It also records per-batch load times in MLflow to help identify data bottlenecks.

from serverless_gpu.data import DataLoader

loader = DataLoader(

    dataset,

    batch_size=32,

    pin_memory=True,

    # num_workers=6, by default

    # prefetch_factor=4, by default

    # raise num_workers to increase parallel reads, or prefetch_factor to deepen each worker’s queue.

)

• Adjust batch size: For large datasets, use a larger batch size so the cost of loading data per batch is spread over more samples, reducing file fetches each step. If GPU memory is tight, pair a larger batch with gradient accumulation to keep the same effective batch size.

• Use Streaming datasets: For very large datasets that do not fit in memory, use streaming approaches.

  • UCVolumeDataset from serverless_gpu.data for streaming files from Unity Catalog volumes with local caching and automatic distributed partitioning.

  • PyTorch IterableDataset for custom streaming logic.

  • Hugging Face datasets with streaming for datasets hosted on the Hub or in volumes.

  • RayData for distributed batch data processing

FAQs

Q: My training run is slow and GPU utilization is low. What should I check first?

A: Start by inspecting the System Metrics tab in MLflow, focusing on GPU utilization, GPU memory usage, and data loading throughput. If GPU utilization is low while the GPU is not memory-bound, your data pipeline is likely the bottleneck. We recommend increasing your batch size or switching to serverless_gpu.data.DataLoader, which is specifically tuned for serverless I/O. Additionally, ensure you are using UCVolumeDataset to cache files locally instead of refetching them from Unity Catalog during every step.

Q: Should I use notebooks or the AIR CLI for my training workload?

A: It depends on your workflow phase. Use notebooks with the @distributed decorator for rapid prototyping and interactive debugging. Switch to the AIR CLI when you need production-grade features like version-controlled YAML configurations, multi-day training runs, or a CI/CD-style approach directly from your IDE.

Q: How do I resume a previous MLflow training run?
A: Set a custom run name using mlflow.start_run(run_name=”your-custom-name”) so you can identify the run later. To resume, call mlflow.start_run(run_id=”<previous-run-id>”) with the earlier run’s ID. This lets you continue logging metrics and artifacts into the same run record.

Q: Which environment should I choose — the base environment or the AI environment?
A: Start with the AI environment (recommended) as it has popular ML frameworks (PyTorch, DeepSpeed, vLLM, Ray, etc.) pre-installed. Use the base environment if you need full control over your dependencies and want a minimal setup. You can also use a base environment defined by your workspace admin, built on organization-standard images. Always pin library versions for reproducibility.

Thanks for reading. Feel free to share any feedback on topics you’d like to see next in the comments section.

Shout out to the AI Runtime PM, Tejas Sundaresan for helping us review the content in this blog.

We plan to work on Part 2 for this AI Runtime blog series next. In this post, we will share some more best practices around remote development options, cost and usage monitoring, and using genie code with AI Runtime. Stay tuned!

Leave a comment

Here are five lessons we learned over a week of benchmarking on Spark 4

Rule 1: Upgrade the Silicon (M8/M9, Not M5/M6)

Before tuning shuffle partitions or threading, pick the right generation.

On the same workload — same Databricks Runtime, same node count, same pipeline — we moved from M6gd to M8gd and saw a rare three-way win:

• 2× throughput (more records per second)

• ~38% lower cost per record processed

• Lower batch latency end to end

Newer Graviton generations (M8, and M9 when available) bring better CPU, memory bandwidth, and local NVMe behavior. That matters most when every micro-batch does shuffle, spill, and multi-table writes.

Practice: Don’t lock onto M5/M6 families because they’re familiar. Benchmark newest generation up on the same job before you optimize code. Often the biggest lever is hardware, not another spark.conf tweak.

Rule 2: Multi-Thread Your foreachBatch Sinks

foreachBatch runs once per micro-batch. If you write to more than one table (main, audit, dead-letter), sequential writes leave the cluster idle between commits.

Practice: Do transforms once, split the batch, submit writes in parallel from the driver.

from concurrent.futures import ThreadPoolExecutor

def process_batch(batch_df, batch_id):
    with ThreadPoolExecutor(max_workers=2) as executor:
        futures = [
            executor.submit(write_to_table_a, batch_df, batch_id),
            executor.submit(write_to_table_b, batch_df, batch_id),
        ]
        for f in futures:
            f.result()

Parallel writes mean more Spark jobs per batch — you’re trading wall-clock time for driver coordination. Profile before raising max_workers.

For a fuller production version of this — caching the batch once, fanning out to N tables instead of 2, propagating thread exceptions properly — see the reference notebook. Two companion videos walk through it: foreachBatch fundamentals first, then parallelizing the writes.

Rule 3: Spill to Local Disk, Don’t Recache What’s Already There

Every task writes its output to the disk on the machine it runs on. You can let that spill hit the network over EBS, or you can pick an instance family with disk physically attached to the node. The attached disk wins on performance, every time.

Default is simple: don't cache. When the fork happens after a shuffle — the batch already passed through a join, aggregation, or stateful dedup — the shuffle write already left a durable copy on local disk that Spark reuses automatically for any second write path, retry or independent action alike (look for a "skipped" stage in the Spark UI), so caching here only duplicates what's already there.

The judgment call lives in the other case: a fork before any shuffle, where a shared read and a few light transforms split straight into two writes with no durable file to fall back on. Skip the cache and both paths redo that shared work from scratch, source read included; cache it and you pay a serialize-then-deserialize round trip through the block manager instead. Which one wins depends on the weight of the shared transforms — a wide UDF or an expensive parse earns the cache, a couple of column selects don’t, and recomputing them twice is cheaper than the round trip.

Here’s the number that makes this rule worth enforcing: memory-optimized (R-series) instances carry at least a ~20% price premium over general-purpose compute — on every major cloud, not just one. Cut the unnecessary caching and you cut the memory pressure that pushes you onto R-series in the first place. That 20% doesn’t vanish — you either bank it as savings or spend it on more CPU. Cache when profiling proves it pays off, not by habit.

Rule 4: Optimize for Finishing on Time, Not Just Finishing

In a latency-bound streaming job, “did the job succeed” is the wrong question. The right one is “did it finish inside the SLA.” A batch that completes in 90 seconds against a 60-second budget hasn’t crashed — it’s succeeded and missed the point.

That reframes how you pick a join strategy. Sort-merge joins are stable: they spill to disk gracefully and rarely blow up outright. But they don’t respond to more hardware as easily — you can’t throw machines at a sort-merge join and expect a proportional speedup. (Full mechanics here.)

Shuffle hash joins do respond to more hardware. They’re less forgiving under memory pressure, but they scale with compute in a way sort-merge doesn’t. On an SLA-bound job, that tunability is worth more than the extra stability. (Full breakdown here.)

Practice: If your streaming job has an SLA attached, treat join strategy as a lever you tune on purpose — not whatever Spark defaults to.

Rule 5: Set shuffle partitions to match cluster cores

Highly discouraged, so unless you absolutely need to, you can just let auto do it for you

The only reason we set this is because we were heavily latency-bound and did not want to leave it to the auto algorithm. That means we were overly conservative.

On join-heavy streams, spark.sql.shuffle.partitions drives task size and scheduler load.

• Too low → fat tasks, spill, OOM

• Too high → tiny tasks, overhead, small files
  

Recommended Companion Reads

Now that you’ve upgraded your hardware and parallelized your sinks, here is how to take your high-performance streaming architecture to the next level:

1. Optimize Your Compute Storage: Moving beyond default node disk settings can dramatically reduce streaming latency. Learn how to Deploy and Manage Local SSDs on AWS or upgrade your block storage setup with GP3 Support on Databricks to instantly maximize write IOPS.

2. Master Advanced Join Tactics: Choosing the wrong join strategy can derail your SLA during transaction spikes. Read our step-by-step guides on How Shuffle Hash Joins Scale and why traditional Sort-Merge Joins Might Limit Your Streaming Performance.

3. Download the Implementation Notebook: Skip the boilerplate. Grab our ready-to-import Parallel ForeachBatch Script from GitHub, and check out the walkthrough videos on ForeachBatch Parallelization (Part 1) and ThreadPool Tuning (Part 2).

FAQ

When should I parallelize foreachBatch writes?

When a single micro-batch writes to multiple independent sinks, such as a main table and a dead-letter path. Parallelization helps when sink writes are a meaningful share of batch duration and the driver can coordinate the extra jobs safely.

Is caching inside foreachBatch usually a good idea?

No. In many streaming pipelines, especially after joins or aggregations, Spark has already materialized shuffle output. Adding .cache() can increase memory use and GC overhead without enough reuse to justify it.

Why does local NVMe matter for streaming jobs?

Because shuffle, spill, and write-heavy micro-batches are sensitive to disk performance. Faster local storage can reduce batch-time variability and improve throughput for join-heavy workloads.

Should I always change spark.sql.shuffle.partitions manually?

No, almost never unless you absolutely can live without them. Auto-optimized settings may be sufficient.

What does “success” mean for a latency-bound streaming job?

Success is not just that the query keeps running. Success means the pipeline consistently finishes each micro-batch within the target latency window and prevents backlog from growing during spikes.

I’ve spent the last 5 years designing and running thousands of streaming jobs. Here’s how I actually see the tradeoff.

1. Zero-Downtime Runtime Upgrades

This is one of the most under-appreciated advantages of SDP. With a classic Structured Streaming job on regular compute, a runtime upgrade looks like this:

1. Stop the job (downtime begins)

2. Update the runtime version on the cluster

3. Restart the job

4. Wait for startup and checkpoint recovery

Every upgrade becomes a scheduled maintenance window. Because of this, teams constantly defer them, accumulate technical debt, and eventually face a forced migration under pressure.

With SDP, your pipelines are versionless. Databricks manages the runtime lifecycle for you. New compute spins up in the background, your pipeline migrates over automatically, and there’s no restart sequence to plan around. If you are running a few dozen streaming pipelines, this alone eliminates an entire category of operational toil.

2. Infrastructure You Don’t Babysit

• Serverless SDP: There is no cluster to configure. SDP auto-scales, provisions, and tears down compute automatically. You write the pipeline logic; the platform handles the rest. No sizing debates, no idle costs between micro-batches, and SLA-backed startup times.

• Classic SDP: If you aren’t on serverless, you still get Enhanced Autoscaling. This feature is purpose-built for streaming, right-sizing your clusters based on actual backlog and throughput rather than rough guesses.

3. Rewind and Recover, Instead of Rebuild

Production pipelines break. Bad data ships, an unexpected code change causes corruption, or a bug slips through code review. On raw Structured Streaming, recovery usually means manually resetting checkpoints or reprocessing everything from scratch.

SDP’s Rewind API changes that story. With a single operation, you can restore a pipeline—including table data, source offsets, and operator state together—to a clean point in time, apply your fix, and resume.

• No checkpoint deletion required.

• No full refreshes needed.

• The question shifts from “How long until we’re caught up?” to “Rewind, fix, resume.”

(Note: This is currently available on request—reach out via Databricks to get access.)

4. Quality, Dependencies, and Schema: Native, Not Bolted On

On raw Structured Streaming, you have to build and maintain operational guardrails yourself. On SDP, they come baked into the platform:

• Data Quality: Declare @dp.expect rules directly in your pipeline code. Choose whether to warn, drop, or fail per rule. Violations are tracked and visualized in the pipeline UI rather than being silently dropped.

@dp.expect("valid_order_id", "order_id IS NOT NULL")
@dp.expect_or_drop("positive_amount", "amount > 0")
@dp.expect_or_fail("no_duplicates", "COUNT(*) = COUNT(DISTINCT order_id)")
def silver_orders():
    return dlt.read_stream("bronze_orders")

• Multi-Table Dependencies: SDP automatically resolves the full DAG (Directed Acyclic Graph) on its own. You don’t need an external orchestrator for pipeline-internal chains.

• Schema Evolution: Unexpected or changed fields automatically route to a rescue column instead of crashing your production run.

• CDC (Change Data Capture): The APPLY CHANGES INTO syntax gives you SCD Type 1/2 out of the box—no hand-written, complex MERGE logic required.

Add a visual pipeline graph UI, a queryable event log, and row-level lineage through Unity Catalog.

5. When Structured Streaming Is Still the Right Call

While you should default to SDP, two situations are worth reconsidering:

• Library Compatibility: Your pipeline depends on third-party or custom libraries that don’t play well with serverless compute environments. Classic compute gives you full control over the environment.

• An Actual SDP Ceiling: You’ve hit a hard, documented limitation for your specific use case, rather than a hypothetical one.

For everything else, choose SDP first.

The Bottom Line

SDP doesn’t replace Structured Streaming—it removes the operational tax most teams end up paying anyway by building this layer poorly by hand. You ship faster, operate with less toil, and keep raw Structured Streaming as an escape hatch for the rare cases that genuinely need lower-level engine control.

If you are evaluating SDP for the first time or weighing whether to migrate existing Structured Streaming pipelines, talk to your Databricks account team. They can walk through your specific architecture and help you identify where the operational wins are real versus theoretical.

FAQ

What is SDP?

SDP stands for Spark Declarative Pipelines. It provides a declarative way to build data pipelines with built-in operations, quality enforcement, dependency management, and observability.

How is SDP different from Structured Streaming?

Structured Streaming is the underlying core engine. SDP builds on top of it, adding production-focused capabilities like managed upgrades, pipeline orchestration, quality rules, and native monitoring.

Is SDP only useful for large teams?

No. Smaller teams often benefit even more because SDP removes the heavy operational lifting that smaller engineering teams don’t have the time or staffing to manage manually.

Is SDP only useful for streaming workloads?

No. While it is exceptionally strong for streaming and incremental pipelines, it also works seamlessly for multi-hop data pipelines that combine both batch and streaming patterns.

Does SDP replace orchestration tools?

For dependencies inside a single pipeline, often yes. SDP manages execution order and downstream dependencies automatically, reducing the need for an external orchestrator within that specific pipeline boundary.

❌ The Anti-Pattern: Full Catalog Scans in a Loop

Here is the code that can causes pipelines to hit metastore rate limits or fail entirely:

# This forces a full scan of the entire metastore catalog on every execution
spark.read.table("catalog.information_schema.tables") \
    .filter((col("table_schema") == my_schema) & (col("table_name") == my_table))

The Technical Bottleneck

The information_schema is built for wide-scope catalog introspection—such as auditing, broad governance mapping, or checking metadata across an entire environment at once.

When you chain a .filter() in Spark after loading information_schema.tables, there is zero predicate pushdown to the underlying metastore database. The metastore is forced to enumerate and return every single object inside that catalog to the Spark engine, leaving Spark to do the actual row filtering client-side in memory.

The Solution: Use True Point-Lookup APIs

If you only need to verify the existence of a single table, you must bypass catalog-wide views and target a single metadata record directly.

Option 1: Programmatic Check via tableExists() (Recommended)

For pure automation pipelines, the native Spark catalog API handles this efficiently without compiling a heavy relational query plan. It hits a single metadata record instantly.

# Target a single metadata record directly
exists = spark.catalog.tableExists("catalog.schema.table")

Option 2: SQL Scoped Check via SHOW TABLES

If you are working strictly within raw SQL strings, limit your query scope using an explicit schema filter and a wildcard match rather than querying the entire catalog.

-- Fast, schema-scoped point lookup
SHOW TABLES IN catalog.schema LIKE 'table_name';

Option 3: Structural Validation via DESCRIBE TABLE

Often, you don’t just need to know if a table exists—you need to ensure its current column layout matches what your engine expects. You can leverage structural evaluation enclosed in an exception block.

from pyspark.sql.utils import AnalysisException

# Validates existence and structure simultaneously 
try:
    spark.sql("DESCRIBE TABLE catalog.schema.table")
    exists = True
except AnalysisException:
    # Catch AnalysisException specifically. A bare 'except' turns 
    # permission errors and transient timeouts into false "table missing" signals.
    exists = False

When Do You Actually Need information_schema?

If your objective is broad discovery—like mapping end-to-end data lineage across the entire workspace—using information_schema is completely appropriate. However, you should follow this single operational rule: Read it exactly once at the top of the job, cache it, and filter from memory.

# Execute exactly one metastore scan total
all_tables = spark.read.table("catalog.information_schema.tables").cache()

# Filter from the cached memory layer as many times as needed
bronze_tables = all_tables.filter(col("table_schema") == "bronze")
silver_tables = all_tables.filter(col("table_schema") == "silver")

The Ultimate Fix: Cut the Check Entirely

Pre-flight existence checks usually exist because a pipeline doesn’t know whether to create a new table or append to an existing one. If you are using Delta Lake, Delta already solves this for you.

Using .mode("append") will automatically create the table on the first write, and seamlessly append to it on every run after:

df.write \
  .format("delta") \
  .mode("append") \
  .saveAsTable("catalog.schema.target_table")

SQL pipelines get the exact same guarantee from a single DDL line at setup:

CREATE TABLE IF NOT EXISTS catalog.schema.target_table (
    id INT,
    data STRING,
    event_time TIMESTAMP
) USING DELTA;

INSERT INTO catalog.schema.target_table SELECT * FROM staging_view;

The Bottom Line: When you see an existence check before a write in a code review, ask what it’s actually guarding against. Usually, there’s no good answer—and removing it will save your metastore from a world of hurt.

And before you accuse me of writing clickbait titles that have little to do with the content of the blog (I would never do that), I assure you that Mbappé will be mentioned no fewer than 4 times.

Subscribe now

The landscape of LLMs evolves so rapidly that it’s nearly impossible to keep up with every model, let alone every dependency change that may be required by them. With AI Runtime and Custom GPU Model Serving, it’s trivially easy to serve the latest OSS models, because we can edit the versions at the outset, just as we would any other notebook.

In honor of their World Cup victory, let’s use a Mistral model: Devstral-Small-2-24B-Instruct-2512, a great mid-sized model for SWE tasks that deploys easily on Databricks custom GPU model serving.

Mbappé.

Most of these package versions are shipped natively with AI Runtime v5, but just to underscore my above point, I’ll set them here as well:

%pip install databricks-sdk==0.102.0 vllm==0.13.0 transformers==4.57.6 mistral_common openai==2.17.0 opencv-python-headless==4.12.* mlflow==3.12.0 hf_transfer==0.1.9
%restart_python

We’ll now set the wd to local disk and configure the endpoint. We set SKIP_LOCAL_SERVE_TEST=True unless we want to load the whole thing into RAM here in the serverless notebook. That would work on a single node H100, but not on an A10. We can also set the max model length here if we don’t need to service long prompts and want to conserve VRAM during serving. We’ll also set the location in UC we want to save this to, the endpoint name, and the sizing here as well.

# Set working directory to local disk (/Workspace doesn't support large files).
import os, tempfile
workdir = tempfile.mkdtemp()
os.chdir(workdir)
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
# Hugging Face model to download.
MODEL_REPO_ID = "mistralai/Devstral-Small-2-24B-Instruct-2512"
ARTIFACTS_PATH = "devstral"        # Local dir the weights download to
SERVED_MODEL_NAME = "devstral"     # Name vLLM exposes the model under.

# Devstral 2 Small FP8 (~24 GB) won't fit an A10
SKIP_LOCAL_SERVE_TEST = True
MAX_MODEL_LEN = 24576
GPU_MEMORY_UTILIZATION = 0.90

# Allowlisted ports for Serverless GPU notebooks are 3000-3999. Model Serving requires 8080.
LOCAL_PORT = 3080
SERVING_PORT = 8080

# Unity Catalog destination
UC_MODEL_NAME = "catalog.schema.devstral_small_2_24b_fp8"

# Serving endpoint configuration.
ENDPOINT_NAME = "austin-devstral-24b-h100"
WORKLOAD_SIZE = "Small"
SCALE_TO_ZERO_ENABLED = False   # GPU_XLARGE/H100 does NOT support scale-to-zero in Beta

from databricks.sdk.service.serving import ServingModelWorkloadType
WORKLOAD_TYPE = ServingModelWorkloadType.GPU_XLARGE

Mbappé.

Now we can download the model:

from huggingface_hub import snapshot_download

snapshot_download(
   repo_id=MODEL_REPO_ID,
   local_dir=ARTIFACTS_PATH,
)

And log it to our custom entry point and print the model URI in case we want to reference that later. This will take a few minutes. Note that we do NOT download from HF in the serving container.

import mlflow
from mlflow.pyfunc.model import ChatModel, ChatCompletionResponse

# Required placeholder. Serving runs the entrypoint, not python_model.predict.
class LLMModel(ChatModel):
   def predict(self, context, messages, params):
       return ChatCompletionResponse.from_dict({"choices": []})

mlflow.set_registry_uri("databricks-uc")

model_info = mlflow.pyfunc.log_model(
   name=SERVED_MODEL_NAME,
   python_model=LLMModel(),
   artifacts={
       "model_dir": ARTIFACTS_PATH,
   },
   metadata={
       "task": "llm/v1/chat",
       "entrypoint": entrypoint(SERVING_PORT),
   },
   extra_pip_requirements=[
       "mlflow==3.12.0",
   ],
)
model_info.model_uri

Now we register the model. Note that env_pack="databricks_model_serving" packages the env and the bundled weights for Serverless Optimized Deployments. This step will take like 15 minutes.

model_version = mlflow.register_model(model_info.model_uri, UC_MODEL_NAME, env_pack="databricks_model_serving")
print(f"Registered {UC_MODEL_NAME} version {model_version.version}")

Mbappé.

Our model is now ready for deployment. This cell provisions its own GPU_XLARGE (i.e. H100) serving endpoint independent of this A10 notebook.

from databricks.sdk import WorkspaceClient
from datetime import timedelta
from databricks.sdk.service.serving import EndpointCoreConfigInput, ServedEntityInput

config = EndpointCoreConfigInput(
   name=ENDPOINT_NAME,
   served_entities=[
       ServedEntityInput(
           entity_name=UC_MODEL_NAME,
           entity_version=str(model_version.version),
           workload_type=WORKLOAD_TYPE,
           workload_size=WORKLOAD_SIZE,
           scale_to_zero_enabled=SCALE_TO_ZERO_ENABLED,
       )
   ]
)

w = WorkspaceClient()
w.serving_endpoints.create_and_wait(name=ENDPOINT_NAME, config=config, timeout=timedelta(minutes=45))

And you’ve done it. With that, all that’s left to do is query the endpoint to confirm it’s working.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import ChatMessage, ChatMessageRole

w = WorkspaceClient()
resp = w.serving_endpoints.query(
   name=ENDPOINT_NAME,
   messages=[ChatMessage(role=ChatMessageRole.USER, content="Hi, will Mbappé lead France to another World Cup victory? And by extension make Mistral the world favorite OSS LLM for SWE tasks?")],
)
print(resp.choices[0].message.content)

Happy coding,

-Austin

Business users do most of their work outside of a data platform: sales reps draft proposals in Outlook, operations leads run their day in Teams, finance models everything in Excel.Yet the answers to most of their questions sit in Databricks: pipeline metrics, supply chain status, support SLAs, financial actuals, etc.

That gap between “where data lives” and “where people work” used to be bridged by reports and dashboards. Now the bridge is AI agents: assistants embedded in Teams, Outlook, Excel, Copilot Hub, Microsoft Foundry projects, or an in-house chat app. The user asks a question in their flow of work. The agent calls back to Databricks. The answer returns with proper governance applied.

The pattern is simple in theory. In practice, the integration involves a few interlocking decisions: which Databricks resources to wire up, how authentication flows, whether the agent supports MCP or only REST, whether users and Databricks share a Microsoft Entra tenant, and what governance controls you need. Get them wrong and you lose all the benefits of governance inherent to Databricks.

This post walks you through a decision tree to get to the right architecture, with the exact set of artifacts to share between the Databricks side and the agent side at the end.

Architecture Components

Before walking the tree, here are some quick definitions of the different Databricks functions we will be dealing with.

Question 1: Which Databricks Resources Need to Be Connected?

Databricks provides four managed MCP servers out of the box. Each connects a different resource type with its own URL pattern and OAuth scope.

All four are governed by Unity Catalog permissions and visible in AI Gateway > MCPs.

Note on Vector Search: The managed MCP server works directly with indexes that use Databricks-managed embeddings; you send plain text and get plain text results. If you use self-managed embeddings, wrap the index in a Model Serving endpoint (for example, via VectorSearchRetrieverTool in a deployed agent) or a UC Function, and call that from the agent instead.

Question 2: How Many Databricks Endpoints, and Does the Agent Support Multiple Tool Endpoints?

This determines whether you connect a single URL or several.

Path A: Single resource

Connect one managed MCP URL directly. Simplest path: no orchestration needed.

Path B: Multiple resources, agent supports multiple endpoints

Create one OAuth application with combined scopes and hand the agent team all the MCP URLs. The same OAuth token works across all of them.

OAuth app scopes (combine as needed):

vector-search genie unity-catalog sql offline_access

Or use all-apis for blanket access (less restrictive).

One OAuth flow, one token, multiple MCP servers. The user authenticates once and the token carries the correct scopes for every resource.

Path C: Multiple resources, agent supports only one endpoint

Two options.

Option 1: Supervisor Agent (recommended for most cases)

Register each Databricks resource as a sub-agent of an AgentBricks Multi-Agent Supervisor Agent. The external agent connects to one Model Serving endpoint.

The external agent connects to a single REST endpoint: POST https://<workspace>/serving-endpoints/<supervisor>/invocations. The supervisor’s LLM handles routing: it reads the user’s query and delegates to the right sub-agent based on descriptions you provide.

Option 2: Custom MCP Server (Databricks App)

Deploy a Databricks App that proxies to multiple managed MCP servers behind a single URL: https://<app-url>/mcp.

Decision rule: Minimal code with LLM-driven routing goes to Supervisor Agent. Lightweight deterministic proxying goes to Custom MCP App.

Question 3: What Authentication Model Is Required?

This is the most consequential decision. It determines how identity flows from the external agent to Databricks.

The key question: does the end user’s identity need to flow through to Databricks?

Yes: On-Behalf-Of (OBO) Authentication

The external agent authenticates on behalf of each individual user. Databricks receives the user’s identity, and Unity Catalog permissions are enforced per-user.

OBO-supported resources on Model Serving:

If you need OBO access to resources outside this list (e.g., UC Volumes), deploy on Databricks Apps instead of Model Serving. Apps support additional OAuth scopes.

No: Service Principal Authentication

The external agent authenticates as a single identity (service principal). All queries run under that SP’s permissions.

Authentication Method Comparison

The question to ask the data security team: do different users need different access levels to the data behind these resources, or is a shared service account acceptable?

Question 4: Are Users and Databricks in the Same Azure Tenant?

This question only matters if you chose OBO or token federation in Q3. With a simple PAT or client credentials inside the same org, skip to Q5.

Same tenant

Standard OAuth flows work directly:

• Users already exist in Databricks via SCIM sync from the same Entra ID

• OAuth authorization code flow authenticates users against the same tenant

• No federation policy needed

Different tenants (cross-tenant)

Common in enterprise scenarios: users live in Tenant A (corporate Entra ID), but Databricks is associated with Tenant B.

Path A: Per-user OBO (users in external tenant need individual identity)

1. Create an account-wide federation policy:

   • Issuer: https://login.microsoftonline.com/<external-tenant-id>/v2.0

   • Subject claim: sub or oid (to match users by their Entra ID object ID)

   • Audience: Databricks account ID or custom value

2. Provision users in Databricks. Federation validates identity but does NOT create accounts. Use SCIM sync from the external tenant, or provision manually.

3. Each user’s Entra ID token is exchanged for a Databricks token carrying their identity.

4. Unity Catalog permissions apply per-user.

Path B: Service Principal (automated agent, no per-user identity)

1. Create a Databricks service principal in your Databricks account.

2. Create a service principal federation policy:

   • Issuer: https://login.microsoftonline.com/<external-tenant-id>/v2.0

   • Subject: The Entra ID app/service principal’s sub or oid claim

   • Audience: Databricks account ID

3. The external agent obtains a JWT from its own Entra ID tenant.

4. Exchanges it via:

POST https://<workspace>/oidc/v1/token

grant_type=urn:ietf:params:oauth:grant-type:token-exchange

subject_token=<entra-id-jwt>

subject_token_type=urn:ietf:params:oauth:token-type:jwt

1. Zero Databricks secrets cross the tenant boundary.

Cross-tenant requirements checklist

Question 5: What Governance Controls Are Needed?

AI Gateway is the enterprise control plane that sits between MCP servers and external agents. It provides the operational controls needed in production.

AI Gateway Capabilities for MCP

Find it in: Workspace sidebar > AI Gateway > MCPs.

Recommended governance setup

1. Rate limiting: set conservative limits initially, increase based on observed usage.

2. Usage tracking: enable from day one to establish a baseline.

3. Audit logging: required for any integration where per-user identity flows through (OBO).

4. IP allowlisting: restrict MCP server access to known external agent IPs.

5. Scope-limited OAuth: use granular scopes (vector-search, genie) instead of all-apis for production.

Question 6: What Information Needs to Be Exchanged Between Teams?

Setting up the integration requires a handshake between the Databricks team and the external agent team. Here is exactly what each side provides.

From the Databricks side

From the external agent team

Setup steps on the Databricks side

1. Get redirect URLs and IPs from the external agent team.

2. Create the OAuth app: Account Console > Settings > App Connections > Add connection.

• Name: descriptive (e.g., foundry-mcp-client)

• Redirect URLs: from agent team

• Client type: as specified by agent team

• Scopes: combine as needed

1. Allowlist IPs in the workspace IP access list (if applicable).

2. Grant UC permissions: ensure users or SPs have access to the vector search index, Genie space, UC functions, or tables.

3. Share credentials: send the agent team the client ID, endpoints, and MCP server URLs.

4. Configure AI Gateway: set up rate limits, enable usage tracking.

Automation-friendly OAuth app creation via CLI:

databricks account custom-app-integration create --json ‘{

“name”: “foundry-mcp-client”,

“redirect_urls”: [”https://<redirect-from-agent-team>”],

“confidential”: true,

“scopes”: [”vector-search”, “genie”, “unity-catalog”, “offline_access”]

}’

The Decision Tree at a Glance

Common Scenarios (Quick Paths)

Closing Thoughts

Working through these six questions before writing any code keeps you from over-engineering a Supervisor Agent when a single MCP URL would have done the job, or from committing to OBO before confirming that your agent framework supports authorization code flow.

If you take only one thing from this post: answer the cross-tenant question early. Tenant topology determines whether you need a federation policy, and federation policies require account admin access on the Databricks side. Discovering that requirement after you have built half the integration is painful.

Done well, none of this complexity ever reaches the user. They open Teams, ask “what was our on-time delivery rate in the Northeast last week,” and get back numbers that respect their permissions, their region, their role. They never see the OAuth flow, the supervisor routing, or the federation policy. They just get the answer where they were already working. That is the whole point.

References

• Use Databricks managed MCP servers

• Connect non-Databricks clients to Databricks MCP servers

• Authentication for AI agents (Model Serving)

• OAuth token federation

• Configure a federation policy

• Supervisor Agent (multi-agent system)

• Host custom MCP servers using Databricks Apps

• Build a multi-agent system on Databricks Apps

• Unity AI Gateway

• Connect agents to unstructured data (Vector Search)

Glossary

Term

Definition

MCP

Model Context Protocol, an open standard for connecting AI agents to tools and data sources

OBO

On-behalf-of authentication: the agent acts with the end user’s identity and permissions

Token Federation

Exchanging an external IdP JWT for a Databricks OAuth token without storing Databricks secrets

AI Gateway

Databricks’ central governance layer for LLM endpoints, MCP servers, and coding agents

Supervisor Agent

A multi-agent orchestrator that routes queries to specialized sub-agents behind a single endpoint

Databricks Apps adoption can outpace the visibility into what’s running, who owns each app, and what it’s costing. A workspace can quickly accumulate hundreds of apps across business teams - some delivering ongoing value, many running 24/7 long after anyone last touched them, with the compute bill climbing alongside.

Effective app management requires two complementary signals:

• System tables : for inventory, ownership, lifecycle events, and cost attribution

• OpenTelemetry export to Unity Catalog : for request-level usage and idle detection [At the time this blog was published, App telemetry is in Public Preview.]

Used together, these answer the full set of questions a platform team needs: what do we have, who owns it, what does it cost, and is anyone using it? This post walks through both layers and shows how to combine them into a daily workflow that includes automated idle-app shutdown.

Pillar 1: Governance and Cost Attribution with System Tables

System tables provide the foundation for Apps governance. Four tables, combined, answer most operational questions:

Questions this layer answers

System tables provide the core signals needed to understand the operational footprint of Databricks Apps across a workspace:

• Ownership : Creation events help identify the original app owner, while ACL change events help surface who currently has management access.

• Cost : Usage data from system.billing.usage, combined with pricing data from system.billing.list_prices, makes it possible to estimate app cost over time.

• Orphaned Apps: Apps with no apparent current manager can be flagged for review as potentially orphaned.

• Deletion history : Delete events provide a historical view of app removals, including timing and actor.

• Recreate cycles: Creation and deletion history can reveal create → delete → recreate patterns that may indicate experimentation, churn, or unstable ownership.

System tables answer what do we have and what does it cost. The next dimension - is anyone actually using it - requires a usage signal.

Pillar 2: Usage Signals with OpenTelemetry

Databricks Apps support native OpenTelemetry export to Unity Catalog. Three telemetry tables can be configured for an app, but they differ in what is required to populate them:

• otel_logs - request logs and platform-emitted auth events. Populated automatically once OTel export is configured and the app is redeployed. No code changes required.

• otel_metrics - request counts, latencies, and other application-level metrics. Only populated after custom instrumentation is added to the app code (e.g. using the OpenTelemetry SDK in the application).

• otel_spans - distributed traces. Only populated after custom instrumentation is added to the app code.

For governance use cases - and idle detection in particular - otel_logs is the primary signal, and the one that works out of the box. Each request lands in otel_logs with an app.name attribute and timestamp, providing per-request activity data that is independent of compute state.

If teams add custom instrumentation later, otel_metrics and otel_spans open up deeper analysis (latency SLOs, error budgets, user journey traces), but they are not required for the workflow described in this post.

Enabling OTel export on an app

Enablement is a single asynchronous update call per app, using the :createUpdate endpoint with an explicit update_mask.

‼️Do not use the direct PATCH /api/2.0/apps/{name} endpoint for this purpose because it performs a full replacement of the app object, which will silently wipe resources, env_vars, and permission attachments when those fields aren’t included in the request body. :createUpdate with update_mask: “telemetry_export_destinations” ensures only that field is modified.

def patch_telemetry(app_name):
   """Enables OpenTelemetry export to Unity Catalog for the specified Databricks App."""
   payload = {
       "app": {
           "telemetry_export_destinations": [
               {"unity_catalog": TELEMETRY_TABLES}
           ]
       },
       "update_mask": "telemetry_export_destinations"
   }
   return requests.post(
       f"{workspace_url}/api/2.0/apps/{app_name}:createUpdate",
       headers=headers,
       json=payload,
   )

Looping this over GET /api/2.0/apps lets you enable telemetry across the full app fleet. After each app is redeployed, otel_logs begins populating automatically once the app receives traffic, while otel_metrics and otel_spans remain empty until the application code is instrumented.

What OTel adds beyond billing data

Both layers are needed for confident operational decisions. An app may be running and accruing DBUs while no user has touched it for weeks.

🚨 Caveats and Operational Considerations with OTel

• Cold-start gap after OTel enablement. The workflow trusts otel_logs to mean ‘this app had no requests in the activity window.’ If OTel export was just enabled, no telemetry has had time to accumulate - every app looks idle, and a first-day run will flag every app for shutdown. Don’t act on the workflow’s recommendations until telemetry has been collecting for at least as long as your idle-detection window.

• PII in telemetry. OTel auto-instrumentation captures request URLs, query parameters, headers, and request bodies. If apps pass user data through URLs, it will land in otel_logs. Apply row-level security on telemetry tables from day one.

• Staggered enablement and redeployment. Patching telemetry config and redeploying causes a user-visible app restart. For workspaces with hundreds of apps, schedule the enablement in off-hours batches and stage tier-1 apps separately to minimize disruption.

• Explicit protection for critical apps. Maintain a PROTECTED_APPS list in the shutdown job. Incident-response tools, on-call dashboards, and tier-1 business apps should be hard-coded as exempt regardless of activity level.

• Telemetry table retention.otel_logs grows quickly at scale. Set retention windows and VACUUM schedules on day one, partition by date, and monitor storage cost. Otherwise, telemetry storage will offset the savings from idle-app shutdown.

Governance using System Tables and OTel

With both System tables and OTel metrics in place, identifying and stopping idle apps becomes a 5-step workflow.

Idle Detection and Shutdown in 5 simple steps

Step 1: Find apps with recent activity

Query the OpenTelemetry logs for any app that has had user requests or auth events in the activity window. A 7-day window is a safe production default; tighten only after observing baseline traffic patterns.

WITH activity AS (
SELECT DISTINCT
  COALESCE(
    service_name,
    variant_get(resource.attributes, '$["app.name"]', 'STRING')
  ) AS app_name
FROM my_catalog.my_schema.otel_logs
WHERE `time` >= current_timestamp() - INTERVAL 7 DAYS
)
SELECT app_name FROM activity WHERE app_name IS NOT NULL

Step 2: Compute idle candidates

Combine the activity set with the live app inventory (GET /api/2.0/apps) and apply three safety filters: only ACTIVE apps are candidates, anything in PROTECTED_APPS is exempt regardless of activity, and any app younger than MIN_AGE_DAYS is skipped so newly-deployed apps have room to ramp.

from datetime import datetime, timezone
from dateutil.parser import parse as parse_ts

PROTECTED_APPS = {
    "oncall-dashboard",
    "incident-router",
    "exec-monitoring",
}
MIN_AGE_DAYS = 7   # grace period for newly-deployed apps

active = {
    row["app_name"]
    for row in spark.sql("<Step 1 SQL>").collect()
}

candidates = []
for app in list_all_apps():
    state = (app.get("compute_status") or {}).get("state")
    age_days = (datetime.now(tz=timezone.utc) - parse_ts(app["create_time"])).days
    if (
        state == "ACTIVE"
        and app["name"] not in active
        and app["name"] not in PROTECTED_APPS
        and age_days >= MIN_AGE_DAYS
    ):
        candidates.append(app["name"])

Step 3: Resolve owners and notify

This step combines both pillars: query system.access.audit (Pillar 1) to derive a per-app ownership table, then resolve owners for the candidate set generated from OpenTelemetry logs (Pillar 2), and send a 7-day warning to each candidate’s owner. The notification mechanism is intentionally a placeholder - Slack, email, or a Databricks Apps inbox notification all work. The audit trail produced in Step 4 is what matters.

# Build the per-app ownership table from System table system.access.audit.
# Run this once during setup; refresh weekly via a separate job.

spark.sql("""
  CREATE OR REPLACE TABLE apps_ownership AS
  WITH created_latest AS (
    SELECT
      get_json_object(request_params['app'], '$.name') AS app_name,
      workspace_id,
      user_identity.email AS creator_email,
      ROW_NUMBER() OVER (
        PARTITION BY get_json_object(request_params['app'], '$.name'),
                     workspace_id
        ORDER BY event_time DESC
      ) AS rn
    FROM system.access.audit
    WHERE service_name = 'apps'
      AND action_name = 'createApp'
      AND event_date >= current_date() - INTERVAL 365 DAYS
  )
  SELECT app_name, workspace_id, creator_email FROM created_latest WHERE rn = 1
""")

# Daily run: scope the ownership lookup to today's candidate set
spark.createDataFrame(
    [(name,) for name in candidates],
    ["app_name"],
).createOrReplaceTempView("idle_app_candidates")

owners = (
    spark.sql("SELECT app_name, creator_email FROM apps_ownership WHERE app_name IN (SELECT app_name FROM idle_app_candidates) AND workspace_id = '<your workspace_id>'")
    .toPandas()
    .set_index("app_name")["creator_email"]
    .to_dict()
)

for name in candidates:
    owner = owners.get(name, ADMIN_FALLBACK_EMAIL)
    notify(  # replace with your Slack / email / paging integration
        to=owner,
        subject=f"App '{name}' scheduled for shutdown in 7 days",
        body=(
            "No activity detected in the last 7 days. "
            "Touch the app or add it to PROTECTED_APPS to keep it running."
        ),
    )

Step 4: Reconcile the journal and enforce the warning period

This step does two things to the journal:

• First, it sweeps stale rows - any journal entry whose app showed activity in the current window is deleted, since the warning’s premise no longer holds. This includes rows from previously-stopped apps whose owners have restarted them, ensuring the reactivation is acknowledged and a fresh warning cycle begins if the app later goes idle again.

• Second, it MERGEs today’s idle candidates into the journal. The actual stop call in Step 5 only fires for apps that have been on the journal for at least WARNING_DAYS and are still idle on the next run, so this guarantees no app is stopped without a full warning period elapsing, and an app that bounces between active and idle starts each new idle cycle with a fresh warning_sent_at.

# Sweep stale journal entries first.
spark.createDataFrame(
    [(n,) for n in active],
    ["app_name"],
).createOrReplaceTempView("active_apps_today")

spark.sql("""
  DELETE FROM shutdown_journal
  WHERE app_name IN (SELECT app_name FROM active_apps_today)
""")

# Then MERGE today's warnings (idle_app_candidates was registered in Step 3).
spark.sql("""
  MERGE INTO shutdown_journal target
  USING (
    SELECT app_name, current_timestamp() AS warning_sent_at
    FROM idle_app_candidates
  ) source
  ON target.app_name = source.app_name
  WHEN NOT MATCHED THEN INSERT (app_name, warning_sent_at, stopped_at, status)
    VALUES (source.app_name, source.warning_sent_at, NULL, NULL)
""")

Step 5: Stop expired candidates with an audit record

For each app that’s been in the journal at least WARNING_DAYS and is still idle and still not protected, call the stop endpoint and write the outcome back to the journal.

‼️ Stop, don’t delete - stopping is reversible, deletion is not.

expired = spark.sql("""
  SELECT app_name
  FROM shutdown_journal
  WHERE warning_sent_at <= current_timestamp() - INTERVAL 7 DAYS
    AND stopped_at IS NULL
""").collect()

for row in expired:
    name = row["app_name"]
    if name in active or name in PROTECTED_APPS:
        continue  # last-second safety check
    r = requests.post(
        f"{workspace_url}/api/2.0/apps/{name}/stop",
        headers=headers,
        timeout=60,
    )
    spark.sql(
        "UPDATE shutdown_journal "
        "SET stopped_at = current_timestamp(), status = :status "
        "WHERE app_name = :name",
        args={"status": str(r.status_code), "name": name},
    )

Orchestration

Wrap the five steps in a single Databricks Job:

• Cadence: daily, off-hours (e.g., 02:00 UTC) so restart impact lands outside business hours

• Identity: a dedicated service principal (apps-governance-sp) with the minimum permissions:

  • SELECT on otel_logs and apps_ownership

  • SELECT / INSERT / UPDATE /DELETE on shutdown_journal

  • CAN_MANAGE on the apps you intend to govern (required to call /stop)

• Parameters: DRY_RUN (default true for the first two weeks of rollout), MIN_AGE_DAYS (default 7), WARNING_DAYS (default 7)

• Alerts: route job failures to the platform-team channel. A silent failure of this job means orphaned compute keeps running - the cost of missed runs is real money.

• Materialization: keep apps_ownership as a Delta table (not a view), refreshed weekly by a separate job. Daily shutdown runs read from the pre-materialized object in milliseconds instead of re-scanning audit logs every time. Apps younger than MIN_AGE_DAYS can’t be candidates anyway, so weekly refresh is sufficient.

Apps monitoring dashboard

The daily shutdown job is one side of the governance loop, an automated action. The other side is decision support: Databricks dashboard that surfaces the same signals to platform reviewers, app owners, and executives.

Built on the same two pillars, an account-wide dashboard answers the questions the platform team gets asked every week:

• Inventory KPIs: total apps, active vs stopped vs error states, OTel enablement coverage

• Cost trend: weekly DBU consumption per app, with hover-to-USD via system.billing.list_prices

• Usage signals: apps used in the last 7 days, apps unused in the last 7 days (the shutdown candidate set), broken down by workspace

• Lineage/ownership: owner email, last access, last admin action - all derived from system.access.audit

Admins see a single account-wide view instead of N per-workspace views. The visual interface turns raw signal into context that lets people decide which apps to keep, retire, or invest in - without manually correlating audit logs, OTel data, and billing tables.

Summary

A complete governance posture for Databricks Apps combines three building blocks:

1. System tables for inventory, ownership, cost, lifecycle, and compliance reporting.

2. OpenTelemetry `otel_logs` export for request-level usage and idle detection - available out of the box once OTel is configured. otel_metrics and otel_spans add deeper application-level observability when teams add custom instrumentation.

3. Automated controls - a daily job that identifies idle apps, notifies owners, and stops stale apps.

System tables answer what do we have, and what does it cost. OpenTelemetry tells you whether anyone is using it. Together, the two enable platform teams to support broad Apps adoption while keeping costs predictable and ownership accountable.

FAQ

1. Why aren’t system tables alone enough to find idle apps?

System tables show inventory, ownership, lifecycle events, and billing, but they do not reliably show whether users are actively interacting with an app. For idle detection, you need request-level activity from telemetry.

2. Which OpenTelemetry table is most important for this workflow?

otel_logs is the key table because it captures request and auth activity without requiring application code changes, once telemetry export is configured and the app is redeployed.

3. Why should the workflow stop apps instead of deleting them?

Stopping is reversible and preserves the app for later use. Deletion is permanent and introduces unnecessary operational risk for an automated governance process.

4. What is the biggest rollout risk when enabling telemetry?

If you run idle detection immediately after enabling telemetry, every app can appear idle because no activity history exists yet. Wait at least as long as your idle window before enforcing shutdown actions.

5. How should critical apps be handled?

Maintain an explicit protected-apps allowlist so business-critical or operational apps are never stopped automatically, regardless of recent activity.

6. How often should the idle-app workflow run?

A daily off-hours run is a practical default. It gives enough frequency to control spend without creating unnecessary operational churn.

• Most production agent failures come from three architectural decisions: data serving path, data access abstraction, and observability.

• Start with the simplest serving path that meets latency needs: use SQL Warehouse first, optimize table layout and queries, then add session caching or OLTP only if measurements justify it.

• Put all data access behind one execution interface like execute_sql(...) so you can centralize routing, caching, retries, parameterization, and safety controls.

• Treat observability as a launch requirement: trace tool calls, store traces in queryable tables, and add a synthetic heartbeat to measure true end-to-end freshness and latency.

• The winning pattern is not “most advanced architecture first”; it is measure, optimize the hot path, and preserve the ability to swap components without rewriting the agent.

The Framework

You can stand up an agent in an afternoon. Standing one that holds latency SLAs under real concurrency is a different exercise — one most teams underestimate.

The good news: most production agents need to talk to data and they will eventually run into the same three forks in the road. Pick well on each, and most of the other problems become tractable. Pick poorly, and you’ll be rebuilding under pressure.

This is a practitioner’s framework for the three decisions. They’re independent — you can tackle them in any order, and most teams end up working on all three in parallel. What matters is that you’ve made a deliberate call on each one.

Decision 1: How will the agent fetch data?

The instinct here is to jump to the most complex serving pattern before tuning the simplest one. Resist it.

You have three realistic options:

• Option A — Direct SQL Warehouse Serverless. Your tool calls go straight to a SQL warehouse. No new infrastructure, no reverse ETL. Worth knowing: Serverless keeps a remote result cache that survives warehouse shutdowns, so repeat queries stay fast even after the warehouse scales to zero. The fastest thing to ship.

• Option B — In-session caching. If your agent makes repeated queries against the same dataset within a single session, fetch it once and cache it in the session so subsequent calls don’t re-hit the warehouse. The implementation can be as simple as a dict or a DataFrame in the agent process, or an embedded SQL engine like DuckDB if you need to issue varied SQL against the cached data. The decision is about recognizing intra-session repetition, not about picking a cache library.

• Option C — Lakebase (OLTP). You reverse-ETL the tables the agent reads into Databricks Lakebase, a managed Postgres, and the agent queries Postgres instead of Delta. Sub-50ms point lookups. Highest performance ceiling, highest operational cost. Lakebase also doubles as a place to store online features for sub-second feature lookups and can persist agent session state — if you’re already running it for those reasons, the marginal cost of using it as your serving layer drops.

The mistake I see most: teams pattern-match on “we need low latency” and jump to Option C before they’ve measured Option A. They end up running a reverse-ETL pipeline they didn’t need.

Here’s the order that actually works:

1. Start with Option A. Get the agent end-to-end on SQL Warehouse/ DBSQL Serverless. Size the warehouse sensibly — Medium for dev, Large for prod is a reasonable starting point — X-Large gets expensive fast and is rarely the right place to begin. Start with min 1, max 3–4 clusters, then watch the query history; if queries are getting queued, add a cluster. Keep the warehouse warm if you can’t tolerate cold-start tail latency on the first query after a scale-down.

2. Tune the simple path before adding a complex one. Did you actually optimize the layout of the table the agent reads? Are your queries hitting the right clustering keys? Are you using native parameterized queries, not inline params? In our experience, half of the “we need OLTP” cases evaporate once the table layout is right.

3. Only then evaluate B or C. The decision turns on access pattern, not vibes. If a single session issues several tool calls that hit the same underlying data, Option B’s in-session caching pays for itself — the trigger is repetition within a session, not session length. If your agent makes single point lookups with hard sub-second SLAs across thousands of concurrent sessions, Option C is the answer.

A note on what actually matters for performance: it’s the engine serving the agent and the layout of the specific table that engine reads — nothing else. Medallion labels are organizational naming, not architecture. The agent doesn’t care what tier a table sits in; it cares whether the table it queries is laid out for fast reads. If your agent reads from a Delta table via SQL Warehouse/DBSQL, that table’s clustering and file layout determine your latency. If your agent reads from Lakebase, the Delta layout upstream is mostly irrelevant; what matters is the Postgres indexing and the sync freshness. Either way: optimize whatever the serving engine actually reads at query time, and let Predictive Optimization handle upstream tables that aren’t in the agent’s hot path. Avoid materialized views with non-deterministic refresh behavior anywhere in that hot path.

For Delta tables specifically, use liquid clustering and put real thought into the clustering keys. Sortable types make great keys — integers, dates, and timestamps in particular, because they have clean min/max statistics that let Delta skip files efficiently. String keys can work, but they’re far more sensitive to encoding choices (lexicographic ordering, padding, casing) and tend to underperform. Pick keys that match the actual filter patterns the agent will hit — usually some tenant or entity ID plus a time dimension — and verify the agent’s queries are pruning files the way you expect by reading the query profile.

Quick tip: If you’re merging into a table continuously — say every 5 seconds — schedule explicit liquid clustering at a regular cadence (every 30–60 minutes). Otherwise you’re relying on Predictive Optimization to catch up on its own schedule, and for hot, agent-facing tables that’s usually not fast enough.

Force yourself to write down, before you build:

• The p95 latency budget per agent tool call, end-to-end, not per-query.

• Average and p90 queries per agent session. If it’s two, the cache barely helps. If it’s ten, it’s a no-brainer.

• The size of the largest object the agent will scan. Some OLTP stores have hard per-instance ceilings; if you’re past them, the architecture has to bend.

Decision 2: How will you expose data to the agent?

Some tool in your agent that needs data —they needs a way to talk to the data plane. The instinct is to give each tool its own SQL connector when needed, its own connection management, its own retry logic. Don’t.

Use one execution interface. A single function — call it execute_sql(query, max_staleness:int = 60) — that every data-bearing tool calls. Routing, caching, parameterization, observability, retries: all internal to that one function. Tools pass SQL, get results back, and don’t know or care whether the query went to SQL Warehouse, an in-memory cache, or Lakebase.

Without abstraction each tool manages its own connector, retries, parameters. With execute_sql One place to route, optimize, and enforce safety. Without the abstraction, every tool reinvents connection management. With it, you change one function to swap engines, tune queries, or harden security.

This is the single highest-leverage abstraction you can put in your agent. Three reasons:

• One place to optimize. Tuning one query path improves every tool simultaneously. Tuning nine separate paths means you’ll tune three and forget the rest.

• One place to swap implementations. When you decide to add caching (Decision 1, Option B) or move to Lakebase (Option C), no tool code changes. You rewrite execute_sql and you’re done.

• One place to enforce safety. Parameterization, query limits, allowed-table lists — all enforced once.

On safety specifically, two things to bake in from day one.

Native parameterized queries, always. If you’re on databricks-sql-connector >= 3.0, use paramstyle=”named”. Inline parameters bypass server-side caching, won’t survive a future connector release, and open up injection vectors.

Keep sensitive identifiers out of the model’s parameter space. If your agent operates on a specific tenant, customer, or dataset, the model should not be the thing choosing which tenant to query. Use a factory/closure pattern: bind the tenant ID into the tool at construction time, so the model can only call the tool — it can’t substitute a different tenant via prompt injection. Save the parameterized version for arguments the model genuinely should control, like a column name to inspect.

def make_detect_missing_data(tenant_id: str):
    @tool
    def detect_missing_data(column_name: str) -> str:
        """Detect null values in a column."""
        return _run(tenant_id, column_name)  # tenant_id closed over
    return detect_missing_data

Compare that to the naive pattern where tenant_id is a tool parameter the LLM fills in. The naive version is one cleverly-worded user message away from a tenant-isolation incident.

Let the caller declare freshness tolerance. execute_sql should accept a max_staleness parameter — the maximum age of cached data the caller will tolerate for this query. If max_staleness is generous (say, 60 seconds), the query runner is free to serve from the in-session cache, the SQL Warehouse remote result cache, or a Lakebase replica with known lag. If it’s zero, the runner bypasses caches and goes to the source of truth. This pushes the consistency-vs-latency tradeoff to the call site, where the tool author actually knows whether eventual consistency is acceptable. Without it, you end up either over-caching (and serving stale answers in places that can’t tolerate them) or under-caching (and paying full warehouse latency for queries that didn’t need fresh data).

Decision 3: How will you know the agent is working?

If you can’t see what your agent is doing in production, you don’t have an agent — you have a black box that occasionally answers questions. Most teams get tracing late, after the first incident. Don’t be that team.

Use MLflow tracing, and route it to Unity Catalog. MLflow’s tracing is OpenTelemetry-compatible. Link your experiment to a Unity Catalog trace location with

syntax trace_location=UnityCatalog(
catalog_name=catalog_name,
schema_name=schema_name,
table_prefix=table_prefix)

and your traces land as Delta tables in UC that you can query with SQL. You’ll thank yourself the first time someone asks “show me every slow trace for tenant X in the last 24 hours.” Without UC-backed traces, that’s an afternoon of grepping logs. With them, it’s a SQL query.

Instrument the right boundaries. At minimum: input validation, tool dispatch, each tool’s SQL execution, and the model call itself. If your agent framework supports autolog, lean on it — but verify it’s actually emitting the spans you expect. Auto-instrumentation has bugs and gets disabled by version mismatches. When autolog isn’t trustworthy in your stack, fall back to manual mlflow.start_span() for the boundaries that matter. Don’t trace everything; trace the things you’d want to filter on during an incident — tenant_id, query_pattern, tool_name.

Add a synthetic heartbeat. Inject a known record at the source on a fixed cadence — every 30 or 60 seconds — and measure when it shows up at the agent. This is the one honest measure of end-to-end latency. System tables and per-stage metrics will tell you each component is healthy while the whole pipeline is silently 90 seconds behind. A heartbeat catches that immediately. It’s about a hundred lines of code; build it before you scale.

Set a latency budget per stage and defend it. Source → ingestion → transformation → serving table → agent. Every stage gets a number. Every stage has an SLO. When something breaks, you know which stage to look at without guessing. The budget also forces an honest design conversation: if your end-to-end target is 30 seconds and CDC alone takes 12, you don’t have 18 seconds for everything else — you have to decide what to cut.

Three observability tactics in one picture: a per-stage budget you can defend, trace spans landing in queryable storage, and a heartbeat that measures end-to-end the same way a user would.

What this looks like in practice

Start simple. Ship Option A for serving. Wrap data access in execute_sql. Turn on MLflow tracing to Unity Catalog and add a heartbeat. Resist the urge to add caching, OLTP, or stateful streaming gymnastics until you’ve measured a real problem.

The teams that ship reliable agents they do it boringly. They benchmark before they scale. They write down their latency budget. They abstract their data access. They trace the things that matter.

The teams that ship unreliable agents skip ahead to the interesting parts and find out, in production, that the boring parts were the load-bearing parts.

You don’t need to pick the optimal architecture on day one. You need to pick one you can swap pieces of without rewriting everything. Get the three decisions above right and you will thank yourself later.

FAQ

1. What is the best default serving option for a new production agent?

Start with SQL Warehouse. It is the fastest path to production, requires the least new infrastructure, and is often sufficient once query patterns and table layout are optimized.

2. When should I add in-session caching?

Add it when a single agent session repeatedly queries the same underlying data. If sessions only issue one or two reads, caching usually adds complexity without much benefit.

3. When does an OLTP serving layer make sense?

Use it when you need very fast point lookups under high concurrency and have hard latency targets that the warehouse path cannot meet even after optimization.

4. Why should every tool call the same execute_sql(...) function?

A shared execution layer gives you one place to manage routing, retries, caching, parameterization, observability, and safety policies. It also makes future architecture changes much easier.

5. What is the biggest security mistake in agent data access?

Letting the model choose sensitive identifiers such as tenant or customer IDs. Those should be bound into the tool outside the model’s control to reduce isolation and prompt-injection risk.

6. What should I instrument first?

At minimum, trace input validation, tool dispatch, SQL execution, and model calls. Then add a synthetic heartbeat so you can measure actual end-to-end lag, not just component-level health.

Check out Databricks native Slack Integration before implementing a custom solution!

https://docs.databricks.com/aws/en/genie-one/genie-slack

Connect multiple Databricks Genie spaces to Slack with per-user authentication, intelligent routing, semantic caching, and audit logging — all deployed as a Databricks App. A production-grade Slack bot that brings Databricks Genie directly into Slack, allowing users to ask natural-language questions across multiple data domains without leaving their workspace.

Special Thanks

I extend my sincere thanks to Sunil Patil (https://www.linkedin.com/in/pppsunil/) for his invaluable help during the brainstorming phase. His insights were instrumental in solving the complex challenge of managing multiple genies across multiple Slack channels. This project definitely levelled up thanks to his help!

Features

• Multi-Genie Support: Route questions to the right Genie space based on channel, thread, AI classification

• Per-User Authentication: Use OAuth 2.0 User-to-Machine with PKCE so queries run under each user’s Unity Catalog permissions

• Threaded Conversations: Maintain Genie context across Slack threads

• Semantic Cache: Reuse answers for similar questions using Databricks Vector Search

• Delta-Aware Cache Invalidation: Automatically invalidate cached answers when underlying Delta tables change

• Audit Logging: Track user activity, cache hits, latency, errors, and Genie usage in Delta tables

• Reliable Genie Calls: Handle rate limits with retry, exponential backoff, and adaptive polling

• Native Deployment: Run the entire application as a Databricks App with no external infrastructure

Architecture

The application runs as a Databricks App and uses two services:

• Slack Bolt in Socket Mode to listen for messages, mentions, button clicks, and threaded replies

• Flask to handle OAuth callbacks and health checks

When a user asks a question in Slack, the bot follows this flow:

1. Checks whether the Slack user has a valid Databricks OBO token

2. Routes the question to the correct Genie space

3. Looks for a semantically similar cached response

4. Validates cached answers against Delta table versions

5. Sends the question to Genie on cache miss

6. Formats the response in Slack with SQL, tables, and suggested follow-ups

7. Writes cache metadata and audit logs to Delta

Prerequisites

1. Databricks workspace

• Active Databricks workspace

• One or more Genie spaces

• SQL Warehouse

• Unity Catalog permissions

• Vector Search endpoint and index

2. Slack App

• Slack workspace with admin access

• Slack app with Socket Mode enabled

• Bot token and app-level token

3. OAuth App

• Databricks OAuth application configured for User-to-Machine authentication

• Redirect URI pointing to the Databricks App callback endpoint

4. Download the application code and make the necessary updates to the config values in app.yaml. Download all files from repo: https://github.com/adgitdemo/ad_databricks/tree/main/multi-genie-multi-slack-channel-app

Setup Instructions

1. Slack App Setup

• Create a Slack app and enable Socket Mode.

• Under OAuth & Permissions, add these bot token scopes:

  • app_mentions:read

  • chat:write

  • im:history

  • im:read

  • im:write

  • channels:history

  • groups:history

  • users:read

• Subscribe to these bot events:

  • app_mention

  • message.im

  • message.channels

  • message.groups

• Install the app to your workspace and save:

  • Bot User OAuth Token

  • App-Level Token

  • Signing Secret

2. Databricks OAuth Setup

• Create a Databricks OAuth application for User-to-Machine authentication.

• Configure the redirect URI to point to the Flask callback route exposed by the Databricks App.

The bot uses OAuth 2.0 with PKCE to securely link each Slack user to their Databricks identity. After authentication, the user’s pending question is automatically processed, so they do not need to re-type it.

3. Databricks Workspace Setup

• Prepare the required Databricks resources:

  • Genie spaces

  • SQL Warehouse

  • Cache Delta table

  • Audit Delta table

  • Vector Search endpoint

  • Vector Search index

  • Catalog and schema permissions

• Permission Grants Summary

4. Configure the Application

• Update the application configuration with:

  • Slack bot token

  • Slack app token

  • Slack signing secret

  • Databricks workspace URL

  • OAuth client ID

  • OAuth redirect URI

  • Genie space mappings

  • Default Genie space alias

  • SQL Warehouse ID

  • Cache table name

  • Audit table name

  • Vector Search endpoint and index

  • Cache similarity threshold

  • Optional Service Principal fallback flag

• Example routing configuration:

{

“trips-data”: {

“space_id”: “your-trips-genie-space-id”,

“description”: “Questions about trips, pickup locations, fares, and distance”

},

“finance”: {

“space_id”: “your-finance-genie-space-id”,

“description”: “Questions about revenue, spend, forecasts, and financial KPIs”

}

}

Usage

In Slack

Direct Messages:

Send a DM to your bot

Ask questions like: “Show me sales data for last month”

Channel Mentions:

Invite the bot to a channel: /invite @YourBotName

Mention the bot: @YourBotName what are the top 10 customers?

Threaded Conversations:

Continue asking questions in a thread

The bot maintains conversation context within threads

Demo Time

In the General Slack Channel, the bot will use the default Genie Space to answer questions.

DM Thread, the bot will use the default Genie Space to answer questions.

In a dedicated Trip Slack Channel, the bot will use the Trip Genie Space to answer questions.

In a dedicated weather Slack Channel, the bot will use the Weather Genie Space to answer questions.

Real-time streaming in the Spark ecosystem (RTM) is a great leap in Spark structured streaming, as it guarantees an end-to-end processing as low as 5 ms

This directly serves use cases where ultra-low latency is crucial for operations, such as financial transactions and infrastructure monitoring, to name a few, which can directly translate to revenue loss or business image if not addressed in timely manners with Databricks’ RTM streaming

In this blog, we will introduce a simple demo notebook(s) focusing on:

• Creating a simulated streaming dataset in a message bus system (Kafka) with control over

  • Programmatically introducing duplicated records in the upstream

  • Queue partitions (As it impacts downstream processing/ patallelism)

  • Rows/Sec for writing into the message bus (Kafka is for demo purposes here but this can be any of the currently supported sources by RTM

• Reading real-time messages from the message bus in Spark real-time streaming mode and writing down to Databricks’ Lakebase serverless instance, including:

  • A simple, direct write (no transformation) using RTM

  • A simple stateful transformation (deduplication)

  • Explicit tuning parameters for both scenarios/use cases in order to quickly evaluate your use case for RTM

How to use this blog/code:

• Real-time mode sizing framework

  • You can change the upstream spec or

  • Downstream specs through the widgets that expose several tuning parameters

• Learning the ropes of a full pipeline, using foreach in a real-time streaming fashion to write down to a JDBC sink (Lakebase Postgres in our demo)

• Clone the code from the GitHub repo to start your journey!

Compute resources used:

• Any supported upstream system by Spark Realtime Mode

  • In this case, we are using a Kafka topic that we can recreate with n partitions (default 8)

• Databricks classic cluster (no autoscaling)

  • Driver: rd-fleet.xlarge (32 GB memory, 4 cores)

  • Workers: 8 rd-fleet.xlarge instances

  • DBR 17.3 LTS+

• Lakebase postgres instance with 2CU’s (Capacity Units)

The flow Illustration:

Important parameters:

In this framework, we included different parameters to control

Records generation (Upstream):

• Duplication of the same record (including the key) across different timestamps, to simulate a duplication happening upstream

• Percentage of duplicate records generated

• Partitions to be created (this will control the slots needed downstream when Spark reads from Kafka upstream in real-time fashion)

Records persistence (Downstream):

• Dedup flag in order to decide a stateful or stateless transformation

• A batching mechanism to increase throughput

Key Code Snippets to use/reuse:

I think the major part that I’d like to expand on is two things, actually:

• defining a Python JDBC-backed code to implement a Foreachwriter

  The structure of this implementation requires at least open, process, and close implementations. Additionally, I added a buffer implementation that is time or row counts based in order to control the throughput
  An implementation skeleton in Python would be as follows (full code available on GitHub)
  

  class PgForeachWriter:
      def open(self, partition_id, epoch_id):
          try:
              self.conn = psycopg2.connect(**conn_kwargs)
              self.conn.autocommit = False
              self.cursor = self.conn.cursor()
              
              # ...
  
              # Buffering state (only used in buffered mode)
              if use_buffered_mode:
                  self.buffer = []
                  self.last_flush_ts = time.time()
              
              return True
          
          except Exception as e:
              # ...
              return False
  
      def _flush_if_needed(self, force=False):
          ...
  
      def process(self, row):
          ...
  
      def close(self, error):
          ...
  
  
  def make_pg_buffered_writer(
      jdbc_url,
      jdbc_user,
      jdbc_password,
      jdbc_driver,
      jdbc_jar_path,
      table_name,
      max_batch_size=100,  # flush when >= this many rows (0 = simple mode)
      flush_secs=2.0,      # flush if last flush older than this (0 = simple mode)
  ):

• After defining the writer class, all you need is to use it in your writeStream with real-time mode, as shown below

  jdbc_writer = make_pg_buffered_writer(
      jdbc_url=JDBC_URL,
      jdbc_user=JDBC_USER,
      jdbc_password=JDBC_PASSWORD,
      jdbc_driver=JDBC_DRIVER,
      table_name=TABLE_NAME,
      max_batch_size=max_batch_size_param,
      flush_secs=flush_secs_param
  )
  
  query = (
      df_for_write
      .writeStream
      .foreach(jdbc_writer)
      .outputMode("update")
      .queryName("jdbc_sink_writer")
      .trigger(realTime="5 minutes")
      .option("checkpointLocation", checkpoint_path)
      .start()
  )

Performance measures:

I ran many scenarios in simulation and consistently sustained around 10k writes/sec over several minutes, with minimal to no delays and no backlog, as shown below. This aligns with a production rate of around 10K upstream writes, which are read in real time by Spark's streaming and written to Lakebase. I was also able to achieve a similar throughput for stateful (deduplication) transformation with no drops

Key considerations:

• RTM mode will handle any recs/sec rate when sized right

• Lakebase will handle writes/sec in the 10’s K range per CU out of the box

• That said, there are tuning parameters and things to consider:

  • To enhance the write throughput, and depending on the use case you can:

    • Consider Lakebase Autoscaling

    • Consider batching the writes to downstream (this is generally best practice) to benefit from Postgres batch DML API’s

    • Batching can be time or number-of-rows-based (example provided in the repo)

  • To enhance the RTM read in general:

    • You can tune your windowing range for stateful transformation

    • Tune the realTime value (default 5 minutes); this parameter is used for checkpoint commit

    • Increase the number of upstream partitions

The sweet spot:

• For the ideal optimum solution:

  • Consider right-sizing of:

    • upstream source and its partitions, as it will control parallelism downstream

    • Spark cluster size makes sure there are enough slots to schedule all stages at once

    • Downstream sizing (in our case, it is Lakebase) and consider autoscaling, more CU’s = More throughput

Conclusion:

While the purpose of this exercise is to demonstrate a full pipeline using Lakebase PostgreSQL as the downstream, and is not benchmarking specifically

The E2E latency in test is P95 356 MS with a P50 of 225 MS

These figures are out of the box with no tuning of any sorts, it can be improved further.

Also, with 8 Kafka topic partitions and an adequate Databricks cluster configuration, there is ~0 lag in real-time streaming read from Kafka and downstream write to Lakebase

That said, please note that your mileage will vary! In essence, that:

• A clear understanding of records production upstream is key

• A correct sizing of the Spark cluster and the Downstream OLTP (Lakebase Postgres in our setup) is crucial for throughput

• There is a trade-off between write capacity and the downstream cluster capacity units

  • Time or number of rows is a good tuning parameter in such a case Or

  • Autoscaling downstream to scale to more throughput requirements should be considered

🙋 Frequently Asked Questions

What is the main goal of this demo?

It demonstrates how to build a low-latency streaming pipeline from Kafka through Spark Real-Time Mode into Lakebase, while exposing the main tuning knobs that affect throughput and lag.

What variables can I change in the simulation?

You can adjust ingest rate, Kafka partition count, duplicate-record percentage, deduplication behavior, and downstream batching settings.

Why does Kafka partition count matter?

Partition count affects how much parallelism Spark can use when reading the stream. Too few partitions can limit throughput even if the cluster has spare capacity.

When should I use buffered writes instead of direct writes?

Buffered writes are usually better when downstream write throughput is the bottleneck. They reduce per-row overhead and improve efficiency for OLTP-style sinks.

Can stateful deduplication still perform well in Real-Time Mode?

Yes — based on our test results, the pipeline achieved similar throughput with deduplication enabled, provided the job was sized appropriately.

What should I tune first in production?

Start with upstream partitions, Spark cluster sizing, downstream write capacity, and batching thresholds. Then refine checkpoint cadence and stateful window settings.

As a Specialist Solutions Architect at Databricks, I often hear the same questions from customers who are migrating dashboards from legacy BI tools to Databricks AI/BI Dashboards:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

In the first blog post in this series, I focused on the first two questions and showed how to recreate:

• context filters using parameters in dataset SQL, and

• “Only Relevant Values” filters using field filters and query-based parameters.

In the second post, I focused on cross-filtering and drill-through interactions.

This third post tackles the remaining question: “How do we do user-based filtering in AI/BI Dashboards?”

In Databricks, these controls live in Unity Catalog, not in the dashboard itself. AI/BI Dashboards query governed tables and views, and Unity Catalog enforces fine-grained access control before the data ever reaches the dashboard. I’ll walk through how to:

• Implement user-based filtering and RLS with dynamic views that use current_user() and is_account_group_member().

• Apply RLS directly on tables using row filters and protect sensitive fields with column masks.

• Scale these patterns across many tables and columns using ABAC tag policies and governed tags.

As in the previous posts, I’ll use the built-in samples.tpch dataset. I’ve also published the companion dashboard so you can import it into your workspace, follow along as you read, and adapt these patterns to your own Unity Catalog data.

1. How User-Based Filtering Maps to Unity Catalog

Before we dive into SQL, it’s useful to clarify where these responsibilities live in Databricks.

In many BI tools, user-specific security is often implemented close to the dashboard or semantic layer:

• You define user- or group-based rules that map principals to specific regions, customers, or business units.

• You may use identity-aware logic in filters or calculated fields.

• You may maintain a security table that drives which slice of data each user can see.

In Databricks, these controls live in Unity Catalog, not in AI/BI Dashboards:

• Object privileges on catalogs, schemas, tables, and views control whether a user can query a given object at all.

• Dynamic views, row filters, and column masks implement row-level security and masking at query time. They can inspect the current user and their groups and return different rows or values per user.

• AI/BI Dashboards simply query those governed tables and views. They never bypass Unity Catalog: any row filters or masks you define apply to every query, regardless of whether it comes from a notebook, Databricks SQL, or an AI/BI dashboard.

The result is conceptually similar to user-based filtering in traditional BI tools, but with one important shift: The security rules live with the data, not with a particular dashboard.

That’s especially important when:

• The same Unity Catalog tables power multiple AI/BI dashboards and external BI tools, and

• You embed AI/BI Dashboards into applications where thousands of users see the same dashboard definition, but each must see a different subset of data.

In the rest of this post, we’ll build up from that idea:

1. Use a dynamic view and a permission table to enforce RLS on a TPCH Sales dataset.

2. Show how to do similar things directly on tables with row filters and column masks.

3. Discuss how to scale those patterns across many tables and columns using ABAC tag policies and governed tags.

2. Building blocks: dynamic views, row filters, and column masks

To implement user-based filtering in Databricks, you really only need three Unity Catalog primitives: dynamic views, row filters, and column masks.

They all rely on the same core idea: At query time, Unity Catalog can look at who is running the query (and which groups they’re in), and then decide which rows and values to return.

2.1 Identity functions

The main functions you’ll use in policies are:

• current_user()
  Returns the current user’s identity (usually their email).

• is_account_group_member(’<group_name>’)
  Returns TRUE if the current user is a member of an account-level group.

You can call these functions from views and from SQL UDFs used by row filters and column masks.

2.2 Dynamic views

A dynamic view is just a normal SQL view whose logic depends on the current user or their groups.

You can:

• Filter rows based on current_user() or group membership.

• Mask or null out columns for certain users.

• Join to a separate permission table that maps users/groups to allowed regions, customers, etc.

Any AI/BI dataset that selects from a dynamic view automatically inherits its logic. You don’t need to add any special configuration in AI/BI itself.

We’ll use a dynamic view for our first TPCH Sales RLS example.

2.3 Row filters

A row filter attaches RLS logic directly to a table, instead of wrapping the table in a view:

• You define a SQL UDF that takes one or more columns as input and returns BOOLEAN.

• You attach it to a table with ALTER TABLE ... SET ROW FILTER ... ON (column[, ...]).

The row filter runs for every query and can call current_user() and is_account_group_member() internally. This is handy when you want:

• A stable table name (no extra view layer), or

• A single table that’s consumed by many tools, all of which should respect the same RLS.

We’ll look at both group-based and current_user() + permission-table examples later in the post.

2.4 Column masks

A column mask is similar, but operates at the column level:

• You define a SQL UDF that returns a “masked” value.

• You attach it with ALTER TABLE ... ALTER COLUMN ... SET MASK ....

This lets you:

• Show full values (for example, email or salary) only to certain groups.

• Show partially masked or null values to everyone else.

Think of it as the Unity Catalog side of “row-level security + column-level masking” that you might combine in legacy BI tools using data source filters and calculated fields.

Next, we’ll put these pieces together in a concrete example: enforcing region-based RLS on a TPCH Sales dataset using a combination of a base view, a permission table, and a dynamic view.

3. Implementing RLS with a dynamic view

Let’s start with a concrete scenario:

• You have a TPCH Sales dashboard shared across multiple sales teams.

• NA Sales Managers should see only AMERICA.

• EMEA Sales Managers should see only EUROPE.

• APAC Sales Managers should see only ASIA.

• Individual users may have their own custom regions.

In many traditional BI tools, you’d typically solve this with a user filter or data source filter that maps groups to Regions, and a security table to keep that mapping up to date.

In Databricks, we’ll use the same logical pattern – but move it into Unity Catalog:

1. Create a base view for TPCH Sales in a demo schema.

2. Create a permission table that maps principals (groups or users) to Regions.

3. Create a dynamic view that joins the base view to the permission table and applies row-level security based on current_user() and is_account_group_member().

4. Create an AI/BI dataset on top of that dynamic view and build a simple table visualization.

Throughout this section, we’ll:

• Read from samples.tpch (which everyone has).

• Create objects in main.demo_tpch (you can substitute another catalog/schema if needed).

All of the CREATE / INSERT statements in this section should be run in a notebook or SQL editor, not inside an AI/BI dataset. In AI/BI, you’ll just SELECT from the resulting view.

3.1 TPCH Sales base view

First, set up a simple demo schema and define a reusable base view for TPCH Sales:

-- Use the default UC catalog and create a demo schema
USE CATALOG main;
CREATE SCHEMA IF NOT EXISTS demo_tpch;

-- Base TPCH Sales view, reading from samples.tpch
CREATE OR REPLACE VIEW main.demo_tpch.tpch_sales_base AS
SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

3.2 Region access permission table

Next, create a permission table that describes who is allowed to see which Region.

-- Optional: you can keep security tables in the same schema
-- or create a separate one, e.g. main.demo_security
CREATE SCHEMA IF NOT EXISTS main.demo_tpch;

CREATE TABLE IF NOT EXISTS main.demo_tpch.tpch_region_access (
  principal_type STRING,   -- 'group' or 'user'
  principal      STRING,   -- group name or user email
  region         STRING    -- must match tpch_sales_base.region
);

INSERT INTO main.demo_tpch.tpch_region_access VALUES
  ('group', 'NA Sales Managers',        'AMERICA'),
  ('group', 'EMEA Sales Managers',      'EUROPE'),
  ('group', 'APAC Sales Managers',      'ASIA'),
  ('user',  'some.user@databricks.com', 'ASIA'),
  ('user',   current_user(),            'AFRICA');

What this does:

• The first three rows grant access based on account-level groups

• The fourth row grants access to ASIA to a specific user, even if they are not in one of those groups.

• The fifth row uses current_user() to grant you, the person running this SQL, access to AFRICA. When you execute the INSERT, Unity Catalog evaluates current_user() to your own email.

If you’re not in any of the NA/EMEA/APAC Sales Managers groups and you’re not some.user@databricks.com, the only applicable rule for you will be the one that says you can see AFRICA. We’ll see the effect of that in a moment when we query through the dynamic view.

3.3 Dynamic view with region-level RLS

Now create a dynamic view that applies row-level security by joining the base TPCH Sales view to the permission table and checking the current user’s identity and groups:

CREATE OR REPLACE VIEW main.demo_tpch.tpch_sales_rls AS
SELECT s.*
FROM   main.demo_tpch.tpch_sales_base AS s
WHERE EXISTS (
  SELECT 1
  FROM   main.demo_tpch.tpch_region_access a
  WHERE  a.region = s.region
    AND (
      (a.principal_type = 'group'
       AND is_account_group_member(a.principal))
      OR
      (a.principal_type = 'user'
       AND a.principal = current_user())
    )
);

This view enforces RLS as follows. For each row in tpch_sales_base, it looks for a matching rule in tpch_region_access based on region and either group or user principle. If no matching rule exists for the current user and that region, the row is filtered out.

You can verify this by running:

SELECT DISTINCT region
FROM main.demo_tpch.tpch_sales_rls;

If you are only granted access via the (’user’, current_user(), ‘AFRICA’) row, you should see: AFRICA.

From now on:

• Any query against main.demo_tpch.tpch_sales_rls returns only the regions granted to the current user.

• This applies uniformly whether the query comes from a notebook, Databricks SQL, or an AI/BI Dashboard.

• You can add or revoke access simply by inserting or deleting rows in main.demo_tpch.tpch_region_access – you don’t need to change the view logic.

3.4 Using the dynamic view in an AI/BI dataset

With main.demo_tpch.tpch_sales_rls in place, using it in AI/BI Dashboards is straightforward. You don’t need to re-implement any RLS logic in AI/BI – the dataset just selects from the governed view.

Create the TPCH Sales (RLS Dynamic View) dataset:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_rls;

Build a simple dashboard page to visualize the dataset. In the companion dashboard, I created a RLS with dynamic view page based on this dataset:

When you view this page, if your only grant is the row we inserted with (’user’, current_user(), ‘AFRICA’), the table will show only rows where the region is AFRICA.

The security logic lives in Unity Catalog (dynamic view + permission table). The dashboard just selects from tpch_sales_rls and automatically respects row-level security for each viewer.

4. Implementing RLS on tables with row filters and column masks

In the previous section, we implemented row-level security for TPCH Sales using a dynamic view and a permission table. That pattern works well when you want a named, shareable view to point AI/BI datasets at.

Unity Catalog also lets you attach RLS and masking logic directly to tables using:

• Row filters – control which rows a user can access in a table.

• Column masks – control what values they see in specific columns.

These policies are evaluated in Unity Catalog at query time and apply to all compute – SQL warehouses, notebooks, and AI/BI Dashboards. Unlike dynamic views, they keep the table name unchanged, which can be important when the same table is shared across many tools.

In this section, we’ll:

1. Create a TPCH Sales table in main.demo_tpch for row filters and masks.

2. Attach a group-based row filter that restricts Regions.

3. Attach a user-based row filter that uses current_user() and a permission table.

4. Add a column mask to protect a sensitive column.

Any AI/BI dataset that selects from this table will automatically respect these policies. You don’t need to configure anything special in AI/BI.

All of the statements below should be run in a notebook or SQL editor. AI/BI Dashboards just query the resulting table.

4.1 TPCH Sales table for filters and masks

Row filters and column masks only apply to tables (and a few other relation types), not to views. To keep things simple, we’ll materialize the tpch_sales_base view from Section 3 into a Delta table that we can attach policies to:

-- Use the same demo catalog and schema as before
USE CATALOG main;
USE SCHEMA demo_tpch;

-- Create a physical table from the base view for row filters and masks
CREATE OR REPLACE TABLE main.demo_tpch.tpch_sales_table AS
SELECT *
FROM main.demo_tpch.tpch_sales_base;

From now on, we’ll attach row filters and masks to main.demo_tpch.tpch_sales_table.
If you point an AI/BI dataset at this table instead of the dynamic view, the behavior will be controlled by these table-level policies.

4.2 Group-based row filter on Region

First, let’s attach a row filter that enforces the same Region rules we used in the dynamic view, but purely based on account-level groups:

-- Row filter function that decides which regions each group can see
CREATE OR REPLACE FUNCTION main.demo_tpch.tpch_region_filter(p_region STRING)
RETURNS BOOLEAN
RETURN
  CASE
    WHEN is_account_group_member('NA Sales Managers')   THEN p_region = 'AMERICA'
    WHEN is_account_group_member('EMEA Sales Managers') THEN p_region = 'EUROPE'
    WHEN is_account_group_member('APAC Sales Managers') THEN p_region = 'ASIA'
    ELSE FALSE
  END;

Attach it to the table:

ALTER TABLE main.demo_tpch.tpch_sales_table
  SET ROW FILTER main.demo_tpch.tpch_region_filter ON (region);

Effect:

• Whenever anyone queries main.demo_tpch.tpch_sales_table, Unity Catalog evaluates tpch_region_filter(region) for each row.

• If the user is in NA Sales Managers, only rows where the region is AMERICA are returned. If they’re in EMEA Sales Managers, they only see EUROPE; APAC Sales Managers see ASIA.

• Users not in any of these groups see no rows from this table.

If you point a dataset at:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_table;

Any visualizations based on such a dataset will now respect the group-based Region logic without going through the dynamic view.

4.3 User-based row filter with current_user() and a permission table

Group-based rules are great for broad roles, but you may need finer control – different users seeing different subsets of customers, accounts, or regions.

We can reuse the same pattern as in Section 3 – current_user() + a permission table – but this time embed it in a row filter function instead of a dynamic view.

Example: restrict access by customer_id:

-- Permission table mapping users to customers they can see
CREATE TABLE IF NOT EXISTS main.demo_tpch.customer_access (
  user_email  STRING,
  customer_id BIGINT
);

-- Example grants
INSERT INTO main.demo_tpch.customer_access VALUES
  ('some.user@databricks.com', 889),
  (current_user(),             1111);  -- Give yourself access

Now create a row filter function that consults this table. To avoid ambiguous name resolution with the customer_id column on the table, we’ll use a parameter name p_customer_id:

CREATE OR REPLACE FUNCTION main.demo_tpch.tpch_customer_filter(p_customer_id BIGINT)
RETURNS BOOLEAN
RETURN EXISTS (
  SELECT 1
  FROM   main.demo_tpch.customer_access a
  WHERE  a.user_email  = current_user()
    AND  a.customer_id = p_customer_id
);

Attach it to the same table. Because a table can have only one row filter, we’ll replace the Region filter from the previous subsection in this example:

ALTER TABLE main.demo_tpch.tpch_sales_table
  DROP ROW FILTER;

ALTER TABLE main.demo_tpch.tpch_sales_table
  SET ROW FILTER main.demo_tpch.tpch_customer_filter ON (customer_id);

Effect:

• Whenever someone queries main.demo_tpch.tpch_sales_table, Unity Catalog evaluates tpch_customer_filter(customer_id) for each row.

• For a given user, only rows whose customer_id appears in main.demo_tpch.customer_access
  for current_user() are returned.

• In the sample data above, you will only see orders for customer 1111, while some.user@databricks.com will see orders for customer 889.

You can verify this quickly:

SELECT DISTINCT customer_id
FROM main.demo_tpch.tpch_sales_table
ORDER BY customer_id
LIMIT 20;

If your only mapping is (current_user(), 1111), this query should return just 1111.

To use this in AI/BI Dashboards, you can point a dataset directly at the table:

SELECT
  region,
  nation,
  customer_id,
  customer_name,
  order_id,
  order_date,
  revenue
FROM main.demo_tpch.tpch_sales_table;

In the companion dashboard, I created a RLS with row filter page based on this dataset:

When you open it, the table and visuals only show data for customers you are allowed to see according to customer_access, without any RLS logic in the dashboard itself.

4.4 Bonus: masking sensitive columns with a column mask

Row filters decide which rows a user can see. Sometimes you also need to partially hide sensitive values within those rows – for example, masking customer names, emails or phone numbers for most users while leaving them fully visible for a small group.

Unity Catalog column masks handle this at the column level using the same pattern: a SQL UDF that can branch on current_user() or group membership.

Suppose we want:

• Users in a PII Full Access group to see full customer names.

• Everyone else to see a partially masked version (for example, just the first few characters).

First, define a masking function:

CREATE OR REPLACE FUNCTION main.demo_tpch.mask_customer_name(name STRING)
RETURNS STRING
RETURN
  CASE
    WHEN is_account_group_member('PII Full Access') THEN name
    ELSE concat(substr(name, 1, 3), '***')
  END;

Attach it as a mask on the table:

ALTER TABLE main.demo_tpch.tpch_sales_table
  ALTER COLUMN customer_name
  SET MASK main.demo_tpch.mask_customer_name;

Effect:

• Users in the PII Full Access group see the full customer_name value.

• All other users see a masked version like Cus*** instead of Customer#000001111.

• The mask is enforced for every query against tpch_sales_table – notebooks, SQL editor, and AI/BI Dashboards – including exports.

Here is what it looks like in the companion dashboard:

5. Scaling RLS with ABAC and governed tags

Everything we’ve done so far (dynamic views, row filters, and column masks) is defined directly on individual Unity Catalog objects. That’s fine for a handful of tables, but it becomes hard to manage when you have dozens of catalogs, hundreds of schemas, and thousands of tables. This is exactly the problem that attribute-based access control (ABAC) with governed tags is designed to solve in Unity Catalog.

At a high level, ABAC adds three building blocks on top of the mechanisms we already used:

• Governed tags – account-level tags like sensitivity, business_domain, or region_scope, with a controlled set of allowed values. You attach these tags to catalogs, schemas, tables, or columns.

• Policy UDFs – reusable SQL UDFs that implement row-filter or column-mask logic, similar to the functions we wrote earlier, but intended to be reused across many datasets.

• ABAC policies – centrally managed policies that say “when a tagged object matches these conditions, apply this row filter or column mask for these principals.” Policies can be attached at the catalog, schema, or table level and inherit down the hierarchy.

Databricks recommends using ABAC as the primary way to apply row filters and column masks at scale, and reserving table-by-table configuration for special cases. Conceptually, you can think of ABAC as “lifting” the patterns from Section 4 into a central policy layer:

1. Tag the data once – apply governed tags to catalogs, schemas, tables, and columns that participate in RLS or masking.

2. Register reusable UDFs – define shared row-filter and mask functions in a governance schema (for example, governance.region_filter() and governance.mask_customer_name()).

3. Create ABAC policies ​​– define policies that attach those UDFs to tagged objects based on tag conditions and target groups.

4. Let tags drive behavior – as new tables and columns are tagged, the appropriate row filters and masks are applied automatically by Unity Catalog.

From an AI/BI Dashboards perspective, the experience is the same as in Sections 3 and 4: the dataset SQL stays simple, and the dashboard filters and visualizations work as usual. The difference is that the security logic is now centralized in ABAC policies and tags instead of being embedded directly into each table or view.

6. Summary and next steps

In many BI tools, user-based filtering and row-level security are often implemented close to the dashboard or semantic layer using user/group mappings, security tables, and identity-aware logic. In Databricks, the key shift is that these controls move into Unity Catalog, and AI/BI Dashboards simply query governed tables and views.

In this post, we walked through three main patterns:

• Dynamic views and permission tables (Section 3)
  We built main.demo_tpch.tpch_sales_rls on top of a base TPCH Sales view and a tpch_region_access table. The dynamic view uses current_user() and is_account_group_member() to return different Regions for different users and groups. AI/BI datasets that query this view automatically inherit the row-level security.

• Table-level row filters and column masks (Section 4)
  We materialized TPCH Sales into main.demo_tpch.tpch_sales_table and attached a row filter that looks up allowed customer_id values in customer_access based on current_user(). We also added a column mask for customer_name, showing full names only to a privileged group. Any dataset that selects from this table sees the combined effect of RLS and masking without any extra logic in the dashboard.

• ABAC and governed tags (Section 5)
  We then zoomed out to show how ABAC can apply the same kinds of row filters and masks at scale, using governed tags, reusable policy UDFs, and central ABAC policies. Instead of configuring each table or view by hand, you tag data once and let policies attach the right filters and masks automatically.

Across all three patterns, the core idea is the same: AI/BI Dashboards stay simple; Unity Catalog enforces who sees which rows and what values.

The companion dashboard brings these ideas together in a concrete, runnable example. If you import it into your workspace and wire it up to the views and tables from this post, you can see exactly how the visuals behave for different users as Unity Catalog applies dynamic views, row filters, and column masks behind the scenes.

Combined with the first two posts in this series (Part 1 and Part 2), you now have a practical set of patterns for implementing filtering, drill-through, and row-level security in Databricks AI/BI Dashboards on top of Unity Catalog.

• The Problem: A “Full Refresh” on Kafka-fed pipelines can cause permanent data loss if older records have aged out of the topic (TTL).

• The Strategy: Use a “Backup-and-Rebase” workflow: archive existing data, identify the last processed offsets, and point the new pipeline to that exact starting position.

• The Execution: This guide demonstrates how to manually configure startingOffsets in Spark to bridge the gap between historical backups and new Kafka data.

• The Result: Seamless streaming table updates with zero data loss or record duplication.

The Constraint: Kafka TTL & Streaming Table Immutability

In Spark Declarative Pipelines (SDP), situations may arise when you need to alter a bronze Streaming Table that is being fed by Apache Kafka. This can pose a challenge as you cannot manually alter Streaming Tables via Alter Table commands.

This is further complicated by the fact that Kafka topics are configured with a finite retention period (Time-to-live or TTL), meaning older records eventually age out. Since a pipeline Full Refresh clears the target Delta Table, you cannot simply update your pipeline definition and full refresh from the source, as older records will be missing. The diagram below illustrates this situation. The Kafka cluster contains user-4 who has already been ingested, and users 5 and 6 who still need to be ingested into our Lakehouse. The older user records (1, 2, and 3) have aged out of the topic’s TTL.

As new records are constantly being appended to this topic, how can I update my pipeline and alter my result table without missing new records (users 5, 6, etc), dropping old records (users 1, 2, and 3), or duplicating records (user-4)? The following example has been fabricated to show the solution. The actual reason for implementing this will vary by use-case.

Example: Ingesting JSON via Kafka

Imagine a pipeline is ingesting JSON data from Kafka. This JSON data contains three high-level fields: name, country, and email. It also contains two nested fields “event” and “device” which contain information about what actions users are taking from which devices.

{”user”:”user-2”,”email”:”user-2@example.com”,
“country”:”CA”,
“device”:{”os”:”android”,”model”:”Pixel 7”,”geo”:{”lat”:39.98,”lon”:-82.98}},
“event”:{”name”:”demo”,”seq”:2,”ts”:”2026-02-17T14:40:26.189Z”}}

This data is being written to a json_bronze table, and is storing the user, email, and country fields as String and the nested fields as Struct types. It also adds Kafka metadata fields.

import dlt
from pyspark.sql.functions import current_timestamp, col, from_json, expr

SERVERS = "REDACTED"
TOPIC = “neil_struct_topic”

# Explicit JSON schema for predictable demos (no schemaLocationKey)
SCHEMA_DDL = “”“
  user STRING,
  email STRING,
  country STRING,
  device STRUCT<os: STRING, model: STRING, geo: STRUCT<lat: DOUBLE, lon: DOUBLE>>,
  event STRUCT<name: STRING, seq: BIGINT, ts: STRING>
“”“

@dlt.table(
    name=”json_bronze”,
    comment=”Raw Kafka payload with explicit JSON schema and rescued data”,
    table_properties={”quality”: “bronze”}
)
def json_bronze():
    df = (
        spark.readStream
            .format(”kafka”)
            .option(”kafka.bootstrap.servers”, SERVERS)
            .option(”kafka.security.protocol”, “SSL”)
            .option(”subscribe”, TOPIC)
            .option(”startingOffsets”, “earliest”)
            .load()
    )

    parsed = (
        df.selectExpr(”CAST(value AS STRING) AS json_str”, “topic”, “partition”, “offset”, “timestamp”)
          .select(
              from_json(
                  col(”json_str”),
                  SCHEMA_DDL,
                  options={”rescuedDataColumn”: “_rescued_data”}  # capture type mismatches/new fields
              ).alias(”data”),
              “topic”, “partition”, “offset”, “timestamp”
          )
          # data.* includes _rescued_data already; do not reselect it to avoid duplicate column error
          .selectExpr(”data.*”, “topic”, “partition”, “offset”, “timestamp AS kafka_timestamp”)
          .withColumn(”ingestion_ts”, current_timestamp())
    )

    return parsed

Below is the current state of our target table. This matches the state of the diagram above. Users 1 through 4 have been ingested into the target Delta table.

Now let’s imagine the nature of our topic changes and the device and event fields need to become flexible, allowing for new nested fields to be added anytime and reflected in our target. In the current implementation, these new fields would not appear automatically in our Struct columns.

One way to allow for flexibility of these nested columns is to update the Struct columns to Variant type. As mentioned above, however, Alter Table is unavailable on a streaming table to update column types. Here’s how to accomplish a streaming table update while ensuring there is no data-loss or duplication.

Step 1: Pause your pipeline

Whether your pipeline is continuous or scheduled to run periodically, you don’t want to be ingesting data while performing these actions.

Under the scheduled Job:

Step 2: Backup Bronze Table

This retains all data we’ve already ingested (users 1-4), including data that no longer exists in our Kafka topic (users 1-3). We’ve now ensured we won’t lose records that have aged out of the source.

CREATE TABLE neil_test_catalog.streaming.json_bronze_backup
SELECT * FROM neil_test_catalog.streaming.json_bronze;

Step 3: Determine Max Offset per Partition

At this point, Kafka has continued to receive new records behind the scenes (user-5 and user-6). We want to ensure that when we Full Refresh our pipeline, we read only these new messages without reprocessing data (user-4). This is achieved by making use of Spark’s readStream startingOffsets parameter.

This parameter allows you to specify which offsets Spark should begin reading data from, the first time a pipeline runs. Keep in mind that in Kafka, offsets are integers that uniquely identify messages per partition, so you’ll have to specify a starting offset for each partition in your topic. SDP uses these offsets to ensure that upon initial startup, the pipeline begins exactly where you intend. From that point forward, Spark continuously records these offsets in its internal checkpoints to track progress over time and guarantee exactly-once processing. It’s also a good idea to store this Kafka metadata in the Delta table itself.

Here’s the same snapshot of our source topic and target Delta table, with offset information included:

Notice that to begin processing at user-5 we will need to specify our pipeline start at offset 4. The startingOffsets parameter expects this information in the following format:

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)

If our topic contained multiple partitions, starting offsets must be set for each partition and would look like this for two partitions:

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4, “1”:6}}’)

But how can we find this information?

Finding via Metadata Columns (If Defined in Pipeline and Tracked in Delta Table)

If you’ve added Kafka metadata to your bronze table, you can retrieve your max offset per partition there. Remember that for this simple example we only have one Kafka partition. If using this method it’s important to note that these results show the most recent offset ingested, and +1 must be added to specify where Spark should start reading.

SELECT partition, MAX(offset)
FROM neil_test_catalog.streaming.json_bronze
GROUP BY partition

Finding via Spark Declarative Pipelines Checkpoints

Another way to retrieve your offset information is to query SDP’s /checkpoints/ folder that your pipeline uses to track state and progress. For more detailed information on checkpoints and how Spark Structured Streaming achieves exactly-once processing, check out this blog: Inside Delta Lake’s Idempotency Magic: The Secret to Exactly-Once Spark.

First, find your streaming table’s storage location via DESCRIBE DETAIL.

Note: If your table is Unity Catalog-managed, this method requires direct read access to the table’s managed storage location.

Using the location, you can append “/_dlt_metadata/checkpoints/your_table_name/” to find the most recent streaming query context (the greatest number). For this streaming table the max result is 8 as shown below.

path = “s3://....”
metadata_path = path + “/_dlt_metadata/checkpoints/neil_test_catalog.streaming.json_bronze/”
display(dbutils.fs.ls(metadata_path))

Within the numbered subfolder under your table name, you will see /offsets/ and /commits/ folders, each also containing numbered folders 0/, 1/, 2/ and so on. These folders represent streaming batches in SDP.

• offsets/N is a write‑ahead log entry written before processing batch N. It stores the end offsets (high‑water mark) for that batch — i.e., “read up to here” for each topic/partition.

• commits/N is only written after batch N has finished successfully.

Because an offsets/N entry can exist even if its corresponding commits/N is missing (a batch started but never committed), you should follow these steps to determine where to retrieve your startingOffsets.

List the batch IDs in both folders:

offsets_path = path + "/_dlt_metadata/checkpoints/your_table_name/8/offsets/"
commits_path = path + "/_dlt_metadata/checkpoints/your_table_name/8/commits/"
display(dbutils.fs.ls(offsets_path))
display(dbutils.fs.ls(commits_path))

Let:

• commit_batches = all numeric batch IDs under commits/

• max_commit_batch = largest value in commit_batches

Use max_commit_batch as the last fully committed batch, and read the matching offsets file:

Always read offsets/max_commit_batch to get the correct startingOffsets JSON.

Ignore any higher batch ID that appears only under offsets/ but not commits/. That batch started but never finished, so if you treat its offsets as your starting position, Spark will behave as if that data has already been read and will skip it rather than processing it.

In my example, both the /offsets/ and /commits/ folders contain only batches 0 and 1, so using max_commit_batch, we read startingOffsets from /offsets/1.

metadata_path = path + “/_dlt_metadata/checkpoints/neil_test_catalog.streaming.json_bronze/8/offsets/1/”
display(dbutils.fs.head(metadata_path))

{”batchWatermarkMs”:0,”batchTimestampMs”:1771439553201,”conf”:{...}}
{”neil_struct_topic”:{”0”:4}}

Notice the final line is our starting partition:offset information in the exact format we created manually above. Spark tracks the next offset to read, so you do not need to increment +1 via this method. Again, if our topic had multiple partitions it might look like:

{”neil_struct_topic”:{”0”:4, “1”:6}}

Step 4: Update pipeline definition

Now it’s time to apply the actual logic changes that prompted this process. For my example, I add variant support to table properties:

table_properties={”quality”: “bronze”, “delta.feature.variantType-preview”: “supported”}

And cast the Struct columns to Variant:

return (
        parsed
            .withColumn(”event”, expr(”parse_json(to_json(event))”))
            .withColumn(”device”, expr(”parse_json(to_json(device))”))
        )

Step 5: Full Refresh Table

With pipeline logic updated, it’s time to run with Full refresh to wipe the target table and ingest the new Kafka records starting at our specified offsets. The resulting table will contain only the records we did not back up via Step 2.

To do this, we run our pipeline with full refresh after adding our startingOffsets (line 28). Here is the final pipeline definition.

.option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)

import dlt
from pyspark.sql.functions import current_timestamp, col, from_json, expr

SERVERS = “REDACTED”
TOPIC = “neil_struct_topic”

# Explicit JSON schema for predictable demos (no schemaLocationKey)
SCHEMA_DDL = “”“
  user STRING,
  email STRING,
  country STRING,
  device STRUCT<os: STRING, model: STRING, geo: STRUCT<lat: DOUBLE, lon: DOUBLE>>,
  event STRUCT<name: STRING, seq: BIGINT, ts: STRING>
“”“

@dlt.table(
    name=”json_bronze”,
    comment=”Raw Kafka payload with explicit JSON schema and rescued data”,
    table_properties={”quality”: “bronze”, “delta.feature.variantType-preview”: “supported”}
)
def json_bronze():
    df = (
        spark.readStream
            .format(”kafka”)
            .option(”kafka.bootstrap.servers”, SERVERS)
            .option(”kafka.security.protocol”, “SSL”)
            .option(”subscribe”, TOPIC)
            .option(”startingOffsets”, ‘{”neil_struct_topic”:{”0”:4}}’)
            .load()
    )

    parsed = (
        df.selectExpr(”CAST(value AS STRING) AS json_str”, “topic”, “partition”, “offset”, “timestamp”)
          .select(
              from_json(
                  col(”json_str”),
                  SCHEMA_DDL,
                  options={”rescuedDataColumn”: “_rescued_data”}  # capture type mismatches/new fields
              ).alias(”data”),
              “topic”, “partition”, “offset”, “timestamp”
          )
          # data.* includes _rescued_data already; do not reselect it to avoid duplicate column error
          .selectExpr(”data.*”, “topic”, “partition”, “offset”, “timestamp AS kafka_timestamp”)
          .withColumn(”ingestion_ts”, current_timestamp())
    )

    return (
        parsed
            .withColumn(”event”, expr(”parse_json(to_json(event))”))
            .withColumn(”device”, expr(”parse_json(to_json(device))”))
        )

On the pipeline page:

Here’s the result: user-5 and user-6 as expected:

Step 6: Insert Historical Records

To complete our intended result table, insert historical data from the backup table, ensuring the data matches the new table format (cast columns, etc.):

INSERT INTO neil_test_catalog.streaming.json_bronze
SELECT user, email, country, to_variant_object(device), to_variant_object(event), _rescued_data, topic, partition, offset, kafka_timestamp, ingestion_ts
FROM neil_test_catalog.streaming.json_bronze_backup

Our final result. Your pipeline may be resumed.

user:string
email:string
country:string
device:variant
event:variant
_rescued_data:string
topic:string
partition:integer
offset:long
kafka_timestamp:timestamp
ingestion_ts:timestamp

Step 7: Revert startingOffsets

To avoid pipeline failures in the case of a future Full Refresh, revert startingOffsets to its prior value.

.option(”startingOffsets”, “earliest”)

Frequently Asked Questions

Q: Why can’t I just run a standard Full Refresh on the pipeline?

A: A Full Refresh clears the target table and re-reads the source from the beginning. If your Kafka topic has a retention policy (TTL), data that has "aged out" of the topic will be permanently lost because it no longer exists in the source to be re-read.

Q: Why is startingOffsets necessary if I have a backup?

A: While the backup saves your history, startingOffsets ensures your pipeline resumes reading exactly where the backup stopped. Without this explicit instruction, the pipeline might default to "earliest" (reading only what remains in Kafka, creating a gap) or "latest" (skipping data that arrived during the maintenance window).

Q: Is this process required for "Append-Only" tables?

A: Generally, yes, if you need to restructure the existing table. If you are only adding new columns that are nullable, you might rely on schema evolution, but fundamental type changes usually require the table to be rewritten.

Back in October I published a blog called Getting Medieval on Token Costs. The code and strategy I provided worked, but as the title implied it was rough around the edges. How rough? Well…

• The API calls to the FMs were synchronous, so QPS would have been Medieval indeed

• The Lakebase instance was provisioned, so it would always be accruing costs even without usage

• There was no UI, so you or your admin would be spending hours fiddling with thousand line SQL queries for enterprise use cases

But no matter, we’ve had a Renaissance!

The repo got three meaningful updates and a handful of smaller ones that collectively move this from being technically functional to something a medium enterprise team might actually want to use on Databricks.

Subscribe now

Quick Refresher

The original solution uses a custom MLflow model serving endpoint as a proxy between your users and whatever foundation model they’re calling. Before the request hits the FM, the endpoint checks two Lakebase tables: one for the user’s token limit and another for how many tokens they’ve already burned through. If they’re over budget, the request ends up like John the Baptist in the cover image. If not, it goes through to the FM and the usage is written back to Lakebase along with the response and remaining balance.

Change 1: Autoscaling Lakebase

The original code used a Provisioned Lakebase instance because that was the only option at the time, but it’s going away and we have something better. Autoscaling Lakebase.

Swapping to Autoscaling Lakebase means you can set minimum and maximum scaling bands, and if you don’t need a high availability instance, it will also allow you to scale to zero during times of no use.

This is the smallest change architecturally, but it’s nice not to pay for compute we don’t need.

Change 2: ResponsesAgent + Async FM Calls

The original code used mlflow.pyfunc.PythonModel and called the FM endpoint via requests.post(), which is synchronous and blocking. Only one request can be handled at a time per unit of concurrency. Which meant the endpoint that was supposed to help you manage costs via budgeting is throttling your throughput instead. While I suppose that is one way to reduce token costs, it’s not very useful.

The new version replaces the PythonModel with a ResponsesAgent and swaps requests for httpx.AsyncClient inside an async def predict_stream(). Now multiple FM calls can be in flight simultaneously and the serving endpoint isn’t waiting on one user’s 20-second Claude response before it can look at the next request in the queue.

The core logic now lives in a standalone rate_limiter_agent.py decoupled from the notebook. The public API is much cleaner:

  agent = TokenRateLimiterAgent(
      db_config={...},
      workspace_client=WorkspaceClient(),
      endpoint_name="ep-your-endpoint",
      group_members={
          "data-science-team": ["andrea@company.com", "john@company.com"],
      },
  )

  # Before calling the FM:
  quota = agent.check_quota("andrea@company.com", "databricks-claude-sonnet-4-5")
  if not quota["allowed"]:
      # Return 429 or block the request
      ...

  # After the FM call completes:
  agent.log_usage(
      user_name="andrea@company.com",
      model_name="databricks-claude-sonnet-4-5",
      prompt_tokens=1200,
      completion_tokens=350,
      request_id="req-abc123",
  )

You can drop this into any existing pipeline without touching the notebook, which certainly helps if you’re integrating this into something that already has its own serving infrastructure.

Change 3: An Actual Frontend

The original had no management UI, which meant you had to set limits by writing manual SQL queries for any new change to your budgeting policy. Not very convenient.

The new repo ships a full Databricks App: a React + FastAPI application that deploys alongside your serving endpoint and gives administrators a no-code interface for setting granular budgets. How granular you ask? Any combination of:

• A user, service principal, or group

• Calling any FM, list of FMs, or across all FMs in the workspace

• That resets every X hours, days, weeks, months, or never

• Limited to a specified count of tokens or dollars

  • Pre-populates token costs from Databricks documentation, but manually editable in case this changes or you have some kind of secret discount I don’t know about

  • This is another nice quality of life feature since tokens are not all created equally; GPT OSS 20B tokens cost about 100x less than GPT 5.4 tokens

The drop-downs auto-populate users, SPs, and groups as well as the Databricks Foundation Models.

It also comes with a handy monitoring dashboard so you can see usage over time, your top consumers, and the most popular models.

The App authenticates to Lakebase via a native Postgres role with a static password stored in Databricks Secrets, so there’s no OAuth token refresh to manage.

An Honest Conclusion

Is this production-grade for an org running thousands of concurrent end users? Maybe not. You might consider mini-batching requests at scale, but there will still be some amount of cost tracking overhead, and this gets more difficult at scale.

Is this production-grade for most actual enterprise teams who want to stop their power users from accidentally burning through their monthly token budget in a week? Yes. I think this solution really shines when you have dozens to hundreds of daily active users who might get greedy on Opus requests without some budget enforcement.

But don’t take my word for it; check it out for yourself. The code lives here and setup instructions are in the README.

Cheers and happy coding.

As a Specialist Solutions Architect at Databricks, I often hear the same questions from customers who are migrating dashboards from legacy BI tools to Databricks AI/BI Dashboards:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

In the first blog post in this series, I focused on the first two questions and showed how to recreate:

• context filters using parameters in dataset SQL, and

• “Only Relevant Values” filters using field filters and query-based parameters.

This blog tackles the third question: “How do we replace filter actions from existing dashboards when we click on a bar/segment/point?”

In many BI tools, this behavior is configured as a filter action (a click on a mark filters other views). In Databricks AI/BI Dashboards, the equivalent interactivity is split into two built-in features:

• Cross-filtering, where clicking a mark in one chart filters other charts on the same page that use the same dataset.

• Drill-through, where right-clicking a mark opens a target page filtered to that selection.

Once you combine cross-filtering and drill-through with the context and cascading patterns, you can reproduce most real-world filter-action workflows.

As before, I’ll use the built-in samples.tpch dataset. I’ve also published the companion dashboard, so you can follow along and inspect the configurations yourself.

Recap: TPCH Sales dataset

To keep examples concrete, we’ll use the TPCH sample data that ships with Databricks in the samples.tpch schema. I’ll reuse the same base dataset, TPCH Sales, from the first blog post, which joins tables region, nation, customer, orders, and lineitem, and computes revenue:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

In the companion dashboard, this is the TPCH Sales dataset. It and its derivatives are used by multiple pages:

• Context filter

• Cascading filters with field filters

• Cascading filters with query-based parameters

• Cross-filtering (new in this post)

• Drill-through details (new in this post)

• and others

Filter actions vs. cross-filtering and drill-through

Before we build anything, it helps to align vocabulary.

In many traditional BI tools, filter actions are configured explicitly:

• You specify one or more source sheets.

• You specify one or more target sheets.

• You choose which fields are passed as filters.

• You choose what happens when the selection is cleared.

In Databricks AI/BI Dashboards, cross-filtering is implicit. You don’t turn it on in the visualization panel.

• It is automatically applied to supported visualization types that use the same dataset.

• When you click a bar, slice, or point, AI/BI adds a filter based on that value and re-runs all other visualizations on the page that share the dataset.

In Databricks AI/BI Dashboards, drill-through is also implicit, but slightly more structured:

• When you right-click a supported chart type, AI/BI shows Drill to → <target page> if there is another page in the dashboard that uses the same dataset.

• The target page opens with all visuals based on that dataset filtered to the selected segment, and any compatible filters on that dataset are auto-populated.

Conceptually:

• Cross-filtering ≈ a within-page filter action.

• Drill-through ≈ a navigation filter action (summary → details).

The rest of the post walks through how to configure your pages so these implicit behaviors “just work”.

1. Recreating within-page filter actions with cross-filtering

A very common dashboard pattern is:

• A summary bar chart (for example, total revenue by nation).

• One or more supporting charts (for example, revenue share by region, or a donut chart for mix).

• A filter action so clicking a bar filters the other visuals.

In AI/BI Dashboards, this becomes cross-filtering on top of the TPCH Sales dataset.

1.1. When cross-filtering is applied

Cross-filtering is applied automatically when all of the following are true:

• The visualizations are on the same page.

• The visualizations use the same dataset (for example, TPCH Sales).

• The visualization type is one of the supported chart types: Bar, Box plot, Heatmap, Histogram, Pie, Scatter, or Point map.

If those conditions are met, there is nothing to enable. Clicks on supported charts become filters for any other visualizations on the page that use that dataset.

1.2. Build the Cross-filtering page

In your AI/BI dashboard, add a page named Cross-filtering.

On this page:

1. Add a bar chart: Revenue by nation

   • Visualization: Bar

   • Dataset = TPCH Sales

   • X axis: nation

   • Y axis: SUM(revenue)

2. Add a pie: Revenue by region

   • Visualization: Pie

   • Dataset = TPCH Sales

   • Slice by (Color): region

   • Value (Angle): SUM(revenue)

3. Add filters (optional)

   • Add a Region filter on TPCH Sales.region

   • Add a Nation filter on TPCH Sales.nation

These are just standard filter widgets. There is no “cross-filtering” toggle anywhere in the configuration.

In the companion dashboard, this page is already built for you.

1.3. Use cross-filtering on the charts

Try the following workflow:

1. Select two regions, AFRICA and ASIA, on the Revenue by region chart.

2. Click the bar for JAPAN in the Revenue by nation chart.

Because cross-filtering is automatic for supported charts that share a dataset:

• AI/BI adds filters Nation: JAPAN and Region: AFRICA, ASIA to the TPCH Sales dataset for this page.

• Both charts get updated accordingly.

You can Reset all to default and try a different filter combination.

From a migration perspective, this answers a common question: “Can clicking a bar automatically update the rest of the dashboard in AI/BI?” Yes – when charts share a dataset and use supported visualization types, cross-filtering works implicitly with no additional configuration.

2. Recreating across-page filter actions with drill-through

Another classic summary → detail filter-action pattern is:

• A summary view (for example, revenue by nation).

• A detail view (for example, individual orders).

• A filter action that passes the selected value into the detail sheet as a filter.

In AI/BI Dashboards, this is implemented as drill-through.

2.1. How drill-through is applied

Drill-through shows up as a right-click option when several conditions are satisfied.

• The source chart is a supported type: Bar, Box plot, Heatmap, Histogram, Pie, Scatter, or Point map.

• There is at least one target page in the same dashboard where:

  • At least one visualization uses the same dataset as the source chart.

  • The field you click on has a compatible filter or column on the target page.

In recent AI/BI releases, drill-through no longer requires an explicit target filter; any visualization based on the same dataset as the source selection is filtered automatically, and filters (if they exist) are populated with the drilled values.

There is no drill-through toggle in the widget side panel. Once the above conditions are true, AI/BI surfaces Drill to → <page name> in the context menu automatically.

2.2. Build the Drill-through details page

In the same dashboard, add another page named Drill-through details.

On this page:

1. Add a detail table visualization

   • Visualization: Table

   • Dataset = TPCH Sales

   • Columns: region, nation, customer_id, customer_name, order_id, order_date, revenue

2. Add filters (optional)

   • Add a Region filter on TPCH Sales.region

   • Add a Nation filter on TPCH Sales.nation

   • Add a Customer filter on TPCH Sales.customer_id

Again, there is no special drill-through configuration here – just a normal page that uses the same dataset.

In the companion dashboard, this page is already built for you.

2.3. Drill from summary to details

Go back to the Cross-filtering page and right-click a bar in the Revenue by nation chart (for example, UNITED STATES).

After clicking Drill to → Drill-through details, AI/BI opens the Drill-through details page and filters all visualizations based on TPCH Sales to Nation: UNITED STATES.

From the user’s perspective, this feels almost identical to a filter action in traditional dashboards that navigates from a summary sheet to a detailed sheet with the selected country carried over.

Cross-filtering vs. drill-through vs. filter widgets

Across this blog and the first blog, you now have three main interaction tools in AI/BI Dashboards:

1. Filter widgets

   • Context filters, cascading filters, and query-based parameters.

   • Best for primary, always-visible controls like Region, Date, Product.

2. Cross-filtering

   • Click data points in a supported visualization to filter other charts on the page that use the same dataset.

   • Best for ad-hoc exploration, answering questions like “What happens if I focus only on this region?” and “Which nations are driving that spike?”.

3. Drill-through

   • Right-click a mark to open another page already filtered to that selection.

   • Best for guided summary-to-detail flows where you don’t want to cram everything onto one page.

A simple migration rule of thumb:

• Use filter widgets to rebuild the core filter panels from your existing dashboards.

• Use cross-filtering to automatically filter other visualizations on the page based on chart interactions.

• Use drill-through to replace “go to sheet” filter actions and connect high-level KPIs to detail pages.

Summary and what’s next

In this second blog post in the series, we answered: “Do you support filter actions when I click on a bar or a point?”

The short answer is yes:

• Cross-filtering lets viewers click on supported charts to filter all other visualizations on the same page that share a dataset – no configuration required.

• Drill-through lets viewers right-click a mark and open another page where visuals on the same dataset are already filtered to the selected values, and any matching filters are pre-populated.

Combined with the patterns from the first blog post – context filters and cascading “Only Relevant Values” filters – you now have a robust toolkit for recreating the interactive filtering experience your users expect from traditional dashboards inside Databricks AI/BI Dashboards.

The companion dashboard now includes: a context filter page, a field-based cascading page, a query-based cascading page, a cross-filtering page, and a drill-through details page. You can import it into your workspace and adapt the patterns to your own datasets.

In the next post in this series, I’ll tackle the remaining migration question: “How do we do user-based filtering and row-level security in AI/BI Dashboards?”

The first section explains how replication from Postgres works, and how to configure it. The second part show how to configure Lakeflow pipelines in your Databricks workspace to ingest from Postgres.

How PostgreSQL logical replication works

PostgreSQL writes every change (insert, update, delete) to the Write-Ahead Log (WAL) before applying it. Logical replication decodes that WAL stream into a human-readable format that consumers can subscribe to.

Three objects are involved:

1. WAL level

Logical replication requires wal_level = logical. On AWS RDS, set rds.logical_replication = 1 in your parameter group (requires a reboot).

2. Publication

A publication is a named filter over which tables to expose for replication. It lives inside a single database.

-- Replicate specific tables
CREATE PUBLICATION mfg_paloalto_cdc_pub FOR TABLE iot1.devices, iot1.alerts;

-- Or replicate all tables in the database
CREATE PUBLICATION mfg_paloalto_cdc_pub FOR ALL TABLES;

One publication per database is typical. If you have multiple PostgreSQL databases on the same server, each needs its own publication.

3. Replication slot

A replication slot is a cursor into the WAL. PostgreSQL retains WAL segments until the slot consumer confirms it has processed them (confirmed_flush_lsn). This guarantees the consumer never misses a change, even if it disconnects temporarily.

SELECT pg_create_logical_replication_slot(’databricks_mfg_paloalto_slot’, ‘pgoutput’);

Key facts about slots:

• Slots are server-wide — the name must be unique across the entire PostgreSQL instance, not just within a database.

• One slot per consumer — a slot can only be consumed by one connection at a time. If a second pipeline tries to use the same slot, it will be rejected.

• Unacknowledged WAL accumulates — if a pipeline is stopped and not consuming, the slot holds back WAL. Monitor pg_replication_slots.lag to avoid disk pressure.

Replication user

Create a dedicated user with replication privileges and read access to the tables being replicated:

CREATE USER databricks_replication WITH PASSWORD ‘...’ REPLICATION;

-- On RDS, also grant the rds_replication role
GRANT rds_replication TO databricks_replication;

GRANT CONNECT ON DATABASE mydb TO databricks_replication;
GRANT USAGE ON SCHEMA iot1 TO databricks_replication;
GRANT SELECT ON ALL TABLES IN SCHEMA iot1 TO databricks_replication;

Replica identity

For CDC updates and deletes to include the old row values (needed to identify which row changed), each table needs a replica identity:

-- Default: uses the primary key (recommended when a PK exists)
ALTER TABLE iot1.devices REPLICA IDENTITY DEFAULT;

-- Full: includes all columns (required when there is no PK)
ALTER TABLE iot1.devices REPLICA IDENTITY FULL;

One slot per consumer, not per catalog

The Databricks docs suggest creating one replication slot per source catalog. A more precise rule is: one replication slot per consumer (ingestion pipeline).

If you want to replicate the same database to two different workspaces, or run multiple pipelines in a test environment, each needs its own slot. Using the same slot across multiple pipelines will cause one of them to lose data.

Multiple databases on the same server

PostgreSQL publications are per-database, but replication slots are server-wide. If you have two databases (mfg_paloalto and mfg_austin) on the same server, create two replication slots with different names.

Server
├── Database: mfg_paloalto
│   └── Publication: mfg_paloalto_cdc_pub        (per-database)
├── Database: mfg_austin
│   └── Publication: mfg_austin_cdc_pub           (per-database)
│
├── Slot: databricks_mfg_paloalto_slot             (server-wide, for mfg_paloalto DB)
└── Slot: databricks_mfg_austin_slot               (server-wide, for mfg_austin DB)

Databricks side

Three entities need to be created in order. Below shows the json sent using the Databricks CLI.

1. Unity Catalog connection

The connection is at the server level — no database name. One connection can serve all databases on the same PostgreSQL instance.

// connection.json
{
  “name”: “my-postgres-connection”,
  “connection_type”: “POSTGRESQL”,
  “options”: {
    “host”: “myinstance.abc123.us-east-1.rds.amazonaws.com”,
    “port”: “5432”,
    “user”: “databricks_replication”,
    “password”: “<password>”
  }
}

databricks connections create --json connection.json

A single connection can be used to ingest from multiple databases, as long as the user has permissions.

2. Gateway pipeline

The gateway pipeline runs continuously on a classic cluster. It connects to PostgreSQL, reads the WAL via the replication slot, and buffers raw CDC events into a storage schema in Unity Catalog.

// gateway.json
{
  “name”: “my-postgres-gateway”,
  “catalog”: “my_catalog”,
  “schema”: “cdc_gateway_storage”,
  “channel”: “CURRENT”,
  “continuous”: true,
  “gateway_definition”: {
    “connection_name”: “my-postgres-connection”,
    “gateway_storage_catalog”: “my_catalog”,
    “gateway_storage_schema”: “cdc_gateway_storage”
  }
  “clusters”: [
    {
      “label”: “default”,
      “driver_node_type_id”: “r5.xlarge”,
      “node_type_id”: “r5.xlarge”,
      “autoscale”: {
        “min_workers”: 1,
        “max_workers”: 5
      }        
    }
  ]
}

Setting min_workers = max_workers = 0 will give you a driver-only cluster, if your CDC stream is low volume.

databricks pipelines create --json gateway.json

Note the pipeline_id returned — it is needed for the ingestion pipeline.

3. Ingestion pipeline

The ingestion pipeline reads buffered events from the gateway and writes them as Delta tables. It runs on serverless and is triggered (not continuous).

source_configurations maps each source database to its replication slot and publication. (Unfortunately the postgres source database is referred to as catalog.source_catalog which is a little confusing.) objects controls which schemas or tables to replicate and where they land.

The json below creates a single pipeline from two source schemas in different databases (mfg_paloalto and mfg_austin). Notice how the slot names correspond to the right database.

// ingestion.json
{
  “name”: “my-postgres-ingestion”,
  “catalog”: “my_catalog”,
  “schema”: “mfg_ingest_metadata”,
  “ingestion_definition”: {
    “ingestion_gateway_id”: “<gateway-pipeline-id>”,
    “source_type”: “POSTGRESQL”,
    “connection_name”: null,
    “objects”: [
      {
        “schema”: {
          “source_catalog”: “mfg_paloalto”,
          “source_schema”: “iot1”,
          “destination_catalog”: “my_catalog”,
          “destination_schema”: “mfg_paloalto_iot1”
        }
      },
      {
        “schema”: {
          “source_catalog”: “mfg_austin”,
          “source_schema”: “iot1”,
          “destination_catalog”: “my_catalog”,
          “destination_schema”: “mfg_austin_iot1”
        }
      }
    ],
    “source_configurations”: [
      {
        “catalog”: {
          “source_catalog”: “mfg_paloalto”,
          “postgres”: {
            “slot_config”: {
              “slot_name”: “databricks_mfg_paloalto_slot”,
              “publication_name”: “mfg_paloalto_cdc_pub”
            }
          }
        }
      },
      {
        “catalog”: {
          “source_catalog”: “mfg_austin”,
          “postgres”: {
            “slot_config”: {
              “slot_name”: “databricks_mfg_austin_slot”,
              “publication_name”: “mfg_austin_cdc_pub”
            }
          }
        }
      }
    ]
  }
}

databricks pipelines create --json ingestion.json

The top level catalog.schema (my_catalog.mfg_ingest_metadata above) hold event logs and checkpoints for all the pipelines.

The above create command also returns a pipeline id ingestion-pipeline-id) which can be used to trigger the pipeline.

Triggering a run

After creation, trigger an initial snapshot + CDC run:

databricks pipelines start-update <ingestion-pipeline-id>

The gateway pipeline should run continually. The ingestion pipeline can be run at whatever interval you want, including continually.

Summary checklist

Summary checklist

Step                    Scope                        Notes
────────────────────────────────────────────────────────────────────────────
wal_level = logical     Server                       Requires reboot on RDS
Replication user        Server                       One user can serve all databases
REPLICA IDENTITY        Per table                    Set before creating the publication
Publication             Per database                 One per database is typical
Replication slot        Per database, server-wide    One slot per ingestion pipeline
                        name
Network access          Server                       Allow Databricks cluster egress IP on port 5432

Today’s data teams face a common challenge: streaming data from applications to their Lakehouse requires maintaining complex infrastructure. The typical setup involves managing a message bus like Kafka, configuring connectors, monitoring pipelines, and dealing with significant operational overhead and costs—all just to move data from point A to point B.

Managed Bus Don’t Solve Everything

While Amazon Managed Streaming for Apache Kafka(Amazon MSK) removes the burden of managing servers, it doesn’t eliminate your responsibility for the message bus itself. You’re still on the hook for capacity planning, topic and partition design, producer and consumer tuning, monitoring and alerting, and upgrade timing. Managed services make these tasks less manual, but upgrades remain risky and the operational complexity persists.

Cost management is another pain point. Amazon Managed Streaming for Apache Kafka(Amazon MSK) bills can balloon quickly due to over-provisioned brokers, excess partitions, high replication factors, and long retention periods. AWS manages the infrastructure, but not your spending discipline—that’s still your problem.

What You Actually Need

A fully abstracted streaming service that eliminates cluster management entirely, letting you focus on building data products instead of babysitting message bus infrastructure.

The Challenge

A rapidly growing organization was processing massive device data volumes from Go applications. After essential first-level processing, they needed to stream data to their data lake for near real-time analytics. To avoid the complexity of managing Kafka or similar message bus infrastructure, they took a shortcut: direct writes to their data warehouse with append-only inserts.

Initially simple, this approach quickly hit walls. As volumes grew, they vertically scaled, then horizontally distributed producers across multiple warehouse instances. Small but relentless queries created network bottlenecks. Excessive delta commits from numerous producers killed throughput. They hit soft limits on connections and write operations. To keep data flowing, they over-provisioned compute—watching costs balloon without proportional gains.

The Solution

Zerobus Ingest provided the purpose-built ingestion layer they needed. Their Go applications integrated the SDK with minimal code changes—same append-only pattern, properly architected. The Write-Ahead Logging (WAL) based system handled buffering and batching, while automatic recovery managed network issues that previously caused data loss.

Data now lands directly in Delta tables, eliminating the warehouse intermediary. The result: lower latency, higher throughput, dramatically reduced costs, and one less system to manage. They got the simplicity of direct writes with the scalability of proper streaming—at a fraction of Kafka’s cost.

Overview

Zerobus Ingest is a fully managed, zero-configuration service that enables record-by-record data ingestion directly into Delta tables. No more intermediate message buses. No more complex configurations. Just point your application at an endpoint and start sending data. The Zerobus Ingest API buffers transmitted data before adding it to a Delta table. This buffering creates an efficient and durable ingestion mechanism that supports a high volume of clients with variable throughput.

Before:

After:

Features

Zerobus Ingest leverages a Write Ahead Log (WAL) architecture that enables it to store and acknowledge accepted records quickly, delivering low write latency for your applications. The system is backed by persistent disk storage where both the Write Ahead Log (WAL) and checkpoints are maintained, enabling several powerful capabilities:

Automatic Recovery - Network issues are handled transparently by the SDK. It automatically reconnects on transient failures and resends unacknowledged records without requiring any application-level error handling code.

Efficient Resource Management - Once data syncs successfully to Delta tables, Zerobus Ingest automatically cleans up Write Ahead Log (WAL) logs and metadata, freeing disk space for new data without manual intervention.

Schema Management - Automatic validation against your Delta table schema catches data quality issues at ingestion time, preventing malformed data from entering your Lakehouse.

SDKs

Users will interact with Zerobus Ingest through a dedicated SDK for their language of choice. The documentation and samples are out for Python SDK, Rust SDK and Java SDK . Both the Go and TypeScript SDKs for Zerobus Ingest are now publicly available. GRPC is the main communication mechanism for Zerobus Ingest.

Databricks documentation contains a well documented guide with sample clients in multiple languages. It guides you right from installing the SDK in your preferred language to creating a Protobuf definition and a sample usage.

Supported Formats

• Protocols: gRPC (primary), HTTP REST, Kafka wire format (coming soon)

• Data Formats: Protocol Buffers, JSON

TIPs

• Visit the table history on UC to get a sense of how frequently the table is updated

• Handle the two exceptions gracefully NonRetriableException, ZerobusException.

• Even though Zerobus Ingest periodically issues data file compactions, so you don’t need to worry about the small files

• Don’t forget to create a table with appropriate data types before you run the client

Components

1. Zerobus Ingest Server - Think of them as scalable stateful pod on K8s attached with an SSD disk(high IOPS). Its responsibilities include:

   • Schema validation of the message to the table.

   • Materializing the data in a timely manner to the target table.

   • Sending an acknowledgement to the client that the data is durable.

2. Smart Networking and orchestration - API proxy which distributes the streams to Zerobus Ingest servers per the target delta table and scales pods as the utilization nears the roof

3. Delta kernel - Record batch writer kicks off every 1-5 seconds, uses Delta kernel(uses Arrow) and writes the record batch to the delta table. Kicks in the PO compaction to avoid small files. Rust APIs hides all the complex details of the Delta protocol specification. Binding available for python.

4. Write-Ahead Log(WAL) - Records are immediately persisted to durable storage(think SSD disks with high IOPS) provided by the cloud platform your databricks is running on and is acknowledged in under 50ms. This guarantees durability even if something fails

Zerobus Ingest fits direct lakehouse writes with durable acknowledgments (no bus-style retention/multi-consumer). Zerobus Ingest it not a replacement of message bus in all scenarios. If you need message bus durability/retention or multiple subscribers, Event Hubs/Kafka is likely a safer choice.

Availability

Databricks only support single availability zone (single AZ) durability. This means Zerobus Ingest service may experience downtime. This might change soon.

Kafka still wins when

Despite the cost advantages of Zerobus Ingest Ingest, Kafka remains a better choice in following scenarios:

Exactly-once semantics requirements - For financial transactions, order processing, or other workflows where duplicate processing could cause serious issues, Kafka’s exactly-once delivery guarantees are critical. While Zerobus Ingest roadmap includes this feature, organizations that need it today must still rely on Kafka.

Ultra-low latency fan-out - If your use case requires multiple consumers reading the same stream with different processing logic, Kafka’s pub-sub model excels. Zerobus Ingest currently lacks the subscriber/consumer model that makes Kafka so powerful for fan-out patterns where one stream feeds multiple downstream applications.

Other Limitations

As of writing,

• Zerobus Ingest provides at-least-once delivery semantics, meaning each message will be delivered one or more times. It does not yet support exactly-once semantics. However, the duplicates can be handled using other Databricks and delta features.

• Zerobus Ingest currently supports writing only to managed Delta tables

• Schema evolution on target tables is not yet supported in Zerobus Ingest, so the table schema must match the incoming message structure.

• Each individual message is limited to a maximum size of 10 MB when processed through Zerobus Ingest.

What’s Next

Databricks is actively enhancing Zerobus Ingest with several key features in development. The roadmap includes exactly-once delivery semantics for stronger consistency guarantees, MQTT protocol support to broaden IoT and device connectivity options, comprehensive CDC pipeline capabilities that will handle updates and deletes in addition to inserts and subscriber/consumer model to enable more flexible data consumption patterns.

Enjoy streaming in a cost efficient and simplified manner!

Zerobus Ingest offers a compelling alternative to message bus in a lot of scenarios. While Kafka remains essential for complex streaming architectures, Zerobus Ingest closes the gap for straightforward ingestion use cases—delivering the reliability you need at a fraction of the cost and complexity.

The cost savings extend beyond infrastructure. Kafka expertise commands premium salaries, and maintaining distributed message bus systems requires dedicated engineering time that could be spent on higher-value work. Zerobus Ingest’s simplicity means junior engineers can manage what previously required highly skilled distributed systems expertise. When you factor in reduced operational overhead, lower training costs, and faster time-to-production, the economics become even more compelling. Sometimes the best architecture isn’t the most sophisticated—it’s the one that solves your problem.

FAQ

What is Zerobus Ingest?

A fully managed ingestion service for sending application data directly into Delta tables without operating a separate message bus.

When is Zerobus Ingest a good fit?

It works best for high-volume, append-only ingestion where the primary goal is to land data in the Lakehouse quickly and simply.

Does Zerobus Ingest replace a message bus for every use case?

No. If you need long retention, multiple downstream subscribers, or mature exactly-once semantics today, a message bus is still the better fit.

What delivery semantics does Zerobus Ingest provide today?

It currently provides at-least-once delivery, so downstream designs should account for possible duplicates.

What tables and formats are supported?

The draft states that Zerobus Ingest currently writes only to managed Delta tables and supports Protocol Buffers and JSON.

What are the main implementation requirements?

You need to create the target table in advance, align the incoming schema with the table schema, and handle SDK exceptions appropri

Reference

Stream 10+ GB/sec to your lakehouse in under 5 seconds with zero infrastructure overhead

Deep dive on Zerobus Ingest, now GA

• The Problem: AI agents generate massive volumes of trace data, but traditional observability tools make that data expensive to retain, difficult to govern, and hard to use in evaluation and analytics workflows.

• The Solution: MLflow now supports writing OpenTelemetry (OTEL) traces directly to Unity Catalog tables via a fully managed, serverless ingestion path.

• The Benefit: By landing traces directly in the Lakehouse, teams get governed, analytics-ready observability data with long-term retention, unified evaluation and monitoring workflows, and no OTEL infrastructure to operate.

• The Outcome: Production traces become immediately usable for analysis and evaluation, enabling faster iteration loops between real-world usage, model evaluation, and continuous improvement.

Why AI Tracing Breaks Traditional Observability

As AI applications move into production, traces become one of the clearest ways to understand how agents actually behave by capturing prompts, tool calls, responses, latency, and execution paths. Without strong tracing, it’s hard to understand why agents behave the way they do, making debugging, evaluation, and governance much more difficult.

The challenge isn’t that observability platforms can’t ingest this data. It’s that AI traces quickly become valuable beyond debugging. Teams want to retain them longer, analyze them with SQL, join them with business and model data, and reuse them for evaluation and monitoring. When traces live only inside observability systems, that flexibility is limited, governance becomes fragmented, and moving data into analytics workflows often requires extra pipelines and duplication, especially when sensitive prompt data is involved.

MLflow and OTEL Trace Ingestion

Databricks now supports writing MLflow traces directly to Unity Catalog using the OpenTelemetry (OTEL) format. In practice, this means traces can be ingested in real time and stored in Delta tables, where they benefit from the same scalability, governance, and tooling as the rest of your data.

This changes how teams can use trace data:

• Real-time ingestion with practical retention: Traces can be written as they’re generated at high throughput (GBs/sec) and retained long-term without the cost pressure typically associated with observability platforms.

• Analyze and govern using the Lakehouse: Once traces are tables, you can treat them like any other dataset: query them with SQL, build dashboards, run ETL pipelines, use tools like Genie, and apply governance controls such as PII masking.

• Use the full MLflow evaluation stack: Persisting traces in Unity Catalog removes typical experiment constraints (such as trace caps), making it easier to run large offline evaluations, monitor production systems, and continuously improve quality as workloads grow.

The Engineering trade-off: SaaS vs. Lakehouse

So why not rely entirely on a SaaS observability tool?

1. Retention economics: Agents generate massive text payloads. Storing this data in Delta Lake on object storage is often significantly more cost-effective than SaaS-based retention models.

2. The PII deadlock: Sending raw prompts to third-party platforms can create InfoSec friction. Keeping traces inside Unity Catalog helps maintain data sovereignty and simplifies governance.

3. Analytics, not just telemetry: SaaS tools are strong for operational metrics like latency, but the Lakehouse gives you something different: an analytics and AI engine. You can join traces with business data — revenue, conversions, customer outcomes — to understand real impact, not just system health. Furthermore, the Lakehouse enables you to apply AI directly to your traces, allowing for advanced use cases like classifying user interactions as ‘good’ or ‘bad,’ and building evaluation frameworks to continuously improve system quality.

Architecture: Serverless OpenTelemetry Ingestion

MLflow tracing can use the OpenTelemetry (OTEL) standard, which separates instrumentation from storage. In a typical OTEL deployment, teams are responsible for running collector fleets, scaling agents, handling backpressure, and managing reliability.

Databricks removes that operational layer by providing a managed OpenTelemetry endpoint, transparently powered by Zerobus. Zerobus is a serverless ingestion engine that enables applications to stream data directly into Delta tables using a gRPC API. Applications can easily export spans, logs, and metrics from any OTEL-compatible client directly to Unity Catalog tables, where the data is stored in Delta format. Zerobus acts as the telemetry pipeline, handling ingestion and durability so teams don’t have to operate their own collectors.

From there, traces become first-class data in the Lakehouse, powering MLflow evaluations and monitoring, ad-hoc SQL analysis, dashboards, and downstream analytics. This creates a continuous improvement flywheel where production behavior feeds evaluation and analysis, which in turn drives faster iteration and better agent performance.

Tutorial: Wiring Traces into the Lakehouse

Sample agent: Support manager assistant

For this blog, we’ll create a simple support manager assistant that we can use to demonstrate tracing end-to-end. The agent can be deployed outside of Databricks, as we’ve done here, highlighting that trace ingestion is decoupled from where the agent runs.

We built a LangGraph agent powered by a Databricks-hosted Claude Sonnet 4 model for reasoning and response generation. The agent calls a Genie Space as a tool, which you can deploy here.

When a user asks a data-driven question, the agent invokes Genie through the MCP tool API. Genie translates the request into SQL, executes it against the support dataset, and returns the result. The agent then summarizes the findings and provides actionable takeaways for a support manager.

Setting up MLflow tracing with UC

Before instrumenting the agent, we first configure MLflow to store traces in Unity Catalog. This involves creating the underlying OpenTelemetry tables and linking them to an MLflow experiment so traces can be searched, analyzed, and annotated from the UI. Start by identifying (or creating) a SQL warehouse and an MLflow experiment, then use the MLflow Python library to create the Unity Catalog tables and link the schema to the experiment. For full steps, follow the docs here.

import os
import mlflow
from mlflow.entities import UCSchemaLocation
from mlflow.tracing.enablement import set_experiment_trace_location

mlflow.set_tracking_uri("databricks")

os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "<warehouse-id>"

experiment_name = "<experiment-name>"
catalog_name = "<catalog>"
schema_name = "<schema>"

experiment_id = mlflow.create_experiment(name=experiment_name)

set_experiment_trace_location(
    location=UCSchemaLocation(
        catalog_name=catalog_name,
        schema_name=schema_name,
    ),
    experiment_id=experiment_id,
)

This setup creates Unity Catalog tables for spans, logs, and metrics. Once traces begin flowing, the MLflow service also creates Databricks views that transform the underlying OpenTelemetry data into an MLflow-friendly format for easier querying and analysis. These include:

• mlflow_experiment_trace_otel_spans: detailed execution steps for each request

• mlflow_experiment_trace_otel_logs: structured events such as metadata, tags, and assessments

• mlflow_experiment_trace_otel_metrics: numerical telemetry captured during execution

• mlflow_experiment_trace_metadata: MLflow tags, metadata, and assessments grouped by trace ID

• mlflow_experiment_trace_unified: a consolidated view that assembles all trace data into a single record per trace. For better performance at scale, consider converting it to a materialized view with incremental refresh.

After configuring the trace destination, agent instrumentation remains the same. You can do automatic and/or manual tracing as described here. In our example, we rely on mlflow.langchain.autolog() to capture the detailed LangGraph execution (model calls and tool calls). We also wrap the entrypoint with @mlflow.trace to establish a request-level root span, allowing each invocation to be observed as a single end-to-end execution.

Inspecting a sample trace

Now that the agent is instrumented and traces are flowing into Unity Catalog, let’s look at a real execution.

For this example, we asked the Support Manager Assistant:

“Which support engineer should I put up for promotion?”

The agent evaluated the request, called the Genie space multiple times to gather supporting data, and returned a recommendation based on performance metrics.

While the response looks straightforward, the trace reveals the underlying execution path that produced it. In the MLflow experiment, we can see each of the tool calls as well as the reasoning logic of our claude sonnet model. We can see that it called the genie space tool three times before putting together a final answer.

We can click through each of the individual steps to study the inputs and outputs.

Because traces are stored as Delta tables, they can be queried like any other dataset. We can start with the mlflow_experiment_trace_unified view, where we will find a record that includes the request, response, trace metadata, and an array of the spans.

Beyond Debugging: Analytics on Trace Data

Now that traces are stored in Unity Catalog, they become immediately available for both batch and streaming analytics.

Governance in Unity Catalog

Prompts and responses, however, often contain sensitive information, so treating trace data as governed data is critical. By storing it in Unity Catalog, traces inherit fine-grained access controls, from catalog and schema permissions to column masking and row-level filtering, enabling secure, production-ready analytics without limiting flexibility.

Once access is established, teams can securely run ad-hoc analytics by querying the underlying tables and views with SQL, as we did above. We can also build ETL pipelines, in addition to dashboards and genie spaces, for actionable business insights.

Dashboards

One of the most powerful aspects of having traces in Unity Catalog is that we aren’t locked into a vendor’s rigid, pre-canned views. Because the traces are in Delta tables, we can build custom dashboards that reflect our specific business logic, not just generic system health.

Using AI/BI Dashboards, we built an AI Operations Center that sits directly on top of our trace tables. This dashboard provides a unified view of our application performance, costs, and reliability. Instead of learning a proprietary query language, we just wrote standard SQL (with the help of AI) to extract exactly what we needed.

Here are some key capabilities this unlocked:

Custom Cost & Token Analysis
Generic “cost” metrics are rarely accurate because every team negotiates different rates or uses fine-tuned models with unique pricing. Since we control the SQL, we embedded our specific pricing logic directly into the query. Our dashboard tracks token usage by model type (e.g., GPT-4o vs. Claude 4 Sonnet) and applies our contract-specific rates to calculate a precise Estimated Cost per Trace. This lets us spot expensive outliers immediately—like a single complex query that costs $0.50 due to a retrieval loop.

Component-Level Performance

High-level latency metrics often hide the real culprit. Is the bottleneck the LLM or is it the Genie space retrieval? We built a “Tool Performance” widget that breaks down latency (P50, P99) and error rates for every individual tool in our agent (e.g., retrieve_docs vs. generate_response). This allows us to pinpoint exactly which step in the chain is degrading the user experience.

Genie spaces

Both business and technical stakeholders often want to explore agent behavior without writing SQL. By exposing trace tables through Genie, teams can enable natural-language analysis over their telemetry data, allowing users to ask questions about performance, tool usage, latency, and model behavior directly. In our example, this could include questions such as:

• What types of requests require escalation?

• Are tool retries increasing?

• Which queries trigger the most complex execution paths?

ETL pipelines

Because traces are stored as Delta tables, they can feed downstream ETL pipelines just like any other dataset. By enabling Change Data Feed (CDF), teams can process trace data incrementally, either in batch or streaming, without repeatedly scanning entire tables.

This makes it possible to operationalize observability. For example, a pipeline could monitor trace patterns and trigger alerts when latency exceeds defined thresholds, tool failures spike, or token usage deviates from expected baselines. These signals can then feed dashboards, notification systems, or automated remediation workflows.

Importantly, this complements real-time protections such as AI Guardrails. While guardrails enforce policy at request time, ETL pipelines create a feedback loop, helping teams analyze trends, refine policies, and continuously improve agent performance.

Closing the Loop: From Production Traces to Evaluation

Once traces are available, they can power the full MLflow 3 evaluation stack, enabling teams to measure, improve, and maintain the quality of their AI applications across the entire lifecycle. Evaluation and monitoring build directly on tracing, allowing the same telemetry captured during development, testing, and production to be scored using LLM judges and custom metrics.

Evaluate during development using AI Judges

MLflow allows us to run evaluations against an evaluation dataset, applying built-in or custom judges to score response quality. One effective approach is to bootstrap this dataset from real traces. Because these prompts originate from actual user interactions, they better represent the scenarios your agent must handle compared to synthetic test cases.

Below, we create an evaluation dataset from recently captured traces. MLflow uses a SQL warehouse to search and materialize dataset records, so be sure to configure the warehouse ID in your environment.

import os
import mlflow
import mlflow.genai.datasets
import time

# Required for dataset operations
os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = MLFLOW_TRACING_SQL_WAREHOUSE_ID

DATASET_NAME = f"{CATALOG_NAME}.{SCHEMA_NAME}.support_management_chatbot_traces"

# Create (or load) the dataset
try:
    eval_dataset = mlflow.genai.datasets.create_dataset(name=DATASET_NAME)
except Exception:
    eval_dataset = mlflow.genai.get_dataset(name=DATASET_NAME)

# Pull recent traces (example - from yesterday)
yesterday = int((time.time() - 60 * 60 * 24) * 1000)

traces_df = mlflow.search_traces(
    filter_string=f"attributes.timestamp_ms > {yesterday}",
    order_by=["attributes.timestamp_ms DESC"],
)

# Merge traces into the dataset
eval_dataset = eval_dataset.merge_records(traces_df[["inputs"]])

With the dataset in place, we can define the judges that will score our application. MLflow provides a set of built-in judges, and also allows us to define custom guidelines tailored to our agent’s expected behavior.

from mlflow.genai.scorers import RelevanceToQuery, Safety, Guidelines

# Define judges
agent_judges = [
    RelevanceToQuery(),
    Guidelines(
        name="analytical_correctness",
        guidelines="The response must correctly interpret the data and avoid unsupported conclusions.",
    ),
    Guidelines(
        name="actionable_support_insights",
        guidelines="The response must provide at least one concrete, data-backed recommendation.",
    ),
    Guidelines(
        name="performance_management",
        guidelines="The response should not recommend admonishing or firing employees.",
    ),
    Safety(),
]

# Run evaluation
eval_results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=predict_fn,
    scorers=agent_judges,
)

eval_results

And we can now see the results in the MLflow experiment.

Production monitoring

Development evaluations help us validate behavior before release, but production monitoring shows us how the application performs with real users. MLflow can automatically evaluate live traces using the same judges, helping us quickly detect regressions, drift, and emerging failure patterns. This turns evaluation from a one-time task into an ongoing practice as the application evolves.

Frequently Asked Questions (FAQ)

• Can I use this for agents running outside of Databricks?

  Yes, the agent can be running anywhere. In fact the support assistant agent example that was used for this blog is deployed locally.

• What are the throughput and storage limits of this solution?

  The ingestion throughput limit is 200 QPS today. There is no limit on storage. Previous limits on traces per experiment are no longer applicable. If you need higher throughput limits, please reach out to your Databricks account team.

• What can I do to ensure my search queries, MLflow experiment experience, and downstream analytics remain performant?

  Consider optimizing the OTEL tables using Z-ordering as described here.

• How does this handle PII found in user prompts?

  This feature does not apply any special handling to PII. However, the data is stored in Unity Catalog, where you can leverage governance capabilities, such as fine-grained access controls, column masking, and row filtering, to manage and restrict downstream access.

Get started

To get started, follow along with the documentation.

Integrate Databricks Genie with Google Chat in 30 mins!

With this integration, your team can query business data directly from Google Chat using natural language. No SQL knowledge needed, no switching between tools.

Features

• Conversational AI in Google Chat - Ask questions like “What were total sales last quarter?” and get instant, data-backed answers

• Rich Card Responses - Results are displayed as formatted Google Chat Cards with data tables, generated SQL

• Response Feedback Loop - Thumbs up/down buttons send ratings back to Genie so space authors can review and improve

• Direct Messages and @Mentions - Works in 1:1 DMs with the bot or by @mentioning it in any Chat Space

• No Infrastructure Required - Runs entirely on Google Apps Script (serverless), no servers to manage, no Databricks App to deploy

• Secure by Default - Credentials stored in Apps Script Properties Service, data governed by Unity Catalog

Architecture

Prerequisites

1. Databricks Workspace

   • Active Databricks workspace

   • Genie space ID

2. Google Workspace with the Enterprise plan

   • Enable Google Chat API in the project

   • Enable App Script in the same project

Setup Instructions

Set up Genie Space in Databricks Workspace and note the few parameters

Genie Details

If you already have a Genie Space, skip to the next step. Otherwise:

1. In your Databricks workspace, navigate to Genie in the left sidebar

2. Click New in the upper-right corner of the screen

3. Add your data sources. Then, click Create

4. From the Settings tab, note the Space ID (a 32-character string) -- you will need this later

Tip: The quality of your Genie Space directly affects the quality of your answers. Add company-specific context, test with expected user questions, and iterate on your table annotations.

Generate a Databricks Token

Apps Script needs a token to authenticate with the Genie API.

1. In your Databricks workspace, click your username (top right) then Settings

2. Go to Developer then Access Tokens

3. Click Generate New Token, give it a description like “Google Chat Genie Bot”

4. Copy the token immediately -- you will not see it again

Also note your workspace URL (e.g., https://your-instance.cloud.databricks.com).

For production: Use OAuth M2M with a service principal instead of a personal access token for better security and automated token rotation.

Google Workspace Setup

Google Cloud Project Setup

1. Go to the Google Cloud Console (console.cloud.google.com)

2. Create a new project (or select an existing one)

3. Navigate to APIs and Services then Library

4. Search for Google Chat API and click Enable

5. Go to APIs and Services then OAuth consent screen

6. Select Internal, fill in the app name (e.g., “Genie Data Bot”), and save

Create the Apps Script Project

1. Go to script.google.com and click New project

2. Rename the project to Genie Chat Bot

3. Download all files from the repo https://github.com/adgitdemo/ad_databricks/tree/main/genie-google-chat-app and create the corresponding .gs files in the Apps Script editor:

• Code.gs -- Chat event handlers (onMessage, onAddedToSpace, submitFeedback)

• appscript.json - App Script configuration

• (Recommended) Go to Project Settings (gear icon) then under the “Script Properties“ section, click Add script property (or Edit script properties):

  • DATABRICKS_TOKEN = your token from Step 1

  • GENIE_SPACE_ID = your space ID from Step 1

  • DATABRICKS_HOST = your workspace URL (no trailing slash)

• The code reads from Script Properties automatically.

Configure the Google Chat API

1. In Apps Script, click Deploy->New Deployment, select Add-On configuration, and deploy. Copy the Deployment ID once process completed.

2. Back in the Google Cloud Console, go to APIs and Services then Google Chat API then Configuration

3. Fill in:

- App name: db-genie-1

- Avatar URL: (optional -- your company logo or Databricks icon)

- Description: Databricks Genie Bot

- Functionality: Check “Join spaces and group conversations”

- Connection settings: Select Apps Script and paste your Head Deployment ID in the Deployment ID field

- Triggers specify the apps script functions to handle interactions

- Visibility: Select “Specific people and groups” and add your team or test users

Add Chat App/Bot App to Google Chat

1. Open the Space

Go into the Space where you want the app.

2. Click the Space name at the top

A menu opens.

3. Choose “Manage apps”

Then click Add apps.

4. Search + Add the app

Usage

Once deployed, your team can interact with the bot in two ways:

Direct Messages -- Open a DM with Genie Data Bot and type your question directly. Great for ad-hoc data exploration.

@Mentions in Spaces -- In any Google Chat Space where the bot is added, type @db-genie-1 followed by your question. Responses appear in the space, and anyone can ask follow-ups in the thread.

Demo Time

Tips and Considerations

• Answer quality depends on your Genie Space. Add detailed table/column descriptions, sample SQL, and company-specific context for best results.

• Sync timeout: Apps Script has a 30-second synchronous response limit for Chat. If your Genie queries take longer, consider an optional async pattern using time-driven triggers and the Advanced Chat Service.

• Rate limits: The Genie API allows approximately 5 queries per minute per workspace during Public Preview.

• Row limits: Genie returns up to 5,000 rows per query. The Chat card displays up to 20 rows for readability, with a link to view full results in Genie.

As a Specialist Solutions Architect at Databricks, I regularly work with customers who are migrating critical analytics from existing BI tools to Databricks AI/BI Dashboards – and the first questions I usually get are about filters.

Teams want to know:

• “What’s the Databricks equivalent of the context filters we use today?”

• “Can we still do cascading filters where each dropdown only shows relevant values?”

• “Do you support filter actions when I click on a bar or a point?”

• “How do we do user-based filtering in AI/BI Dashboards?”

These aren’t cosmetic features. They’re how analysts actually interact with dashboards, and they’re often the reason an existing BI dashboard feels “alive” instead of static.

In this post, I’ll walk through how to implement two familiar filter patterns from existing BI dashboards in Databricks AI/BI Dashboards, using the built-in samples.tpch dataset:

1. Context filters → implemented as parameters in dataset SQL

2. “Only Relevant Values” or cascading filters → implemented with field filters and query-based parameters

Row-level security and user-based filtering deserve their own deep dive, and action-style interactions (cross-filtering and drill-through) could easily fill another post, so I’ll cover those separately.

I’ve also published the companion dashboard, so you can follow along and inspect the configurations yourself.

Quick primer: datasets and filters in Databricks AI/BI Dashboards

Before we map those patterns, it helps to align on a few AI/BI Dashboards concepts:

Datasets

In AI/BI Dashboards, each dashboard has a Data tab where you define one or more datasets:

• A dataset is defined by an SQL query, direct reference to a Unity Catalog table/view, or an uploaded file.

• Multiple visualizations can reuse the same dataset.

• Datasets are bundled with the dashboard when you share/export/import it.

Practically, a dataset is your “model” for a set of visuals: one query, many charts.

Field filters vs parameter filters

AI/BI Dashboards support two core ways to filter data from a dashboard: field filters and parameter filters. Both are implemented as filter widgets, but they behave differently under the hood.

Field filters are applied directly to dataset fields (columns) on top of the dataset query. Processing behaviour is defined by the dataset performance thresholds. Specifically, for small datasets (≤ 100K rows or ≤ 100MB), results are pulled to the browser and visualization-specific filtering and aggregation are applied client-side. For larger datasets, Databricks wraps the dataset query in a WITH clause and applies the filter predicates and aggregations in Databricks SQL warehouse (DBSQL).

Parameter filters are applied to parameters, which are variables that get substituted into your dataset SQL at runtime. When the parameter value changes, the query is always re-run in DBSQL.

In other words, field filters operate on the results of the dataset query, while parameter filters operate inside the dataset SQL itself.

To speed up processing, various caching layers in AI/BI Dashboards and DBSQL are used.

We’ll use parameter filters to emulate context filters, and field filters + query-based parameters to emulate “Only Relevant Values.”

Filter scope: global, page-level, and widget-level

Filters in AI/BI Dashboards also differ by scope:

• Global filters are interactive filters in the global filters panel that apply across all pages of the dashboard to any visualization that shares the selected datasets.

• Page-level filters are interactive filter widgets placed on a specific page in the canvas. They apply to all visualizations on that page that share one or more datasets.

• Widget-level filters are static filters configured directly on a single visualization widget in its configuration panel. Authors set the values, and viewers can’t change them.

With that foundation in place, we can now map these context filters and “Only Relevant Values” patterns into AI/BI Dashboards patterns.

Sample dataset: TPCH on Databricks

To keep examples concrete, we’ll use the TPCH sample data that ships with Databricks in the samples.tpch schema.

For the purposes of this post, you can start by creating a dataset that joins tables region, nation, customer, orders, and lineitem:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

In AI/BI Dashboards, you define this query as a dataset in the Data tab and then reuse it across multiple visualizations. Let’s call this dataset TPCH Sales.

We’ll reuse this same dataset or its derivatives throughout the rest of the post to illustrate context filters and cascading filters.

Implementing context filters with parameters in dataset SQL

What a context filter does

A context filter defines a high-level subset of the data:

• The context filter is applied first, often materializing a temporary subset.

• Other filters and some calculations are then evaluated on top of that subset.

Context filters are used to:

• Improve performance by filtering early and shrinking the working set.

• Enforce logical order, such as “always filter by Region first.”

• Make other filters depend on that subset.

How to think about context in AI/BI Dashboards

Given the primer:

• Field filters operate on the results of the dataset query (Databricks wraps your dataset SQL and applies them on top).

• Parameter filters substitute values directly into your dataset SQL, so they filter inside the query, before joins and aggregations.

If you want “context” behavior – filter first, then apply everything else – you should implement that filter as a parameter in the dataset SQL, driven by a parameter filter widget.

Pattern: treat the context as a base parameter

Let’s add a context filter for Region:

If you’re following along with the companion dashboard, this setup lives on the “Context filter” page.

Step 1. Define TPCH Sales (Context) with a Region parameter

Create a dataset TPCH Sales (Context):

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey
WHERE r.r_name = :region_param      -- “context” filter

In the dataset’s Parameters panel:

• Define region_param with type String.

• Optionally set a default (for example, AMERICA) so the dataset runs without any dashboard filter.

This makes region_param the context for all visuals that use TPCH Sales (Context).

Step 2. (Optional but nice) Create a helper dataset for Region values

You can drive region_param directly from TPCH Sales (Context), but a tiny helper dataset keeps things tidy, convenient, and re-usable.

Create TPCH Regions (Context):

SELECT DISTINCT r_name AS region
FROM samples.tpch.region
ORDER BY region;

This dataset has no parameters; it just returns the list of available regions.

Step 3. Add a Region parameter filter widget

We will configure the widget as a page-level filter (alternatively, you can move it into the global filters panel if it should apply across pages).

On the page where you want Region as a context filter:

1. Add a filter widget and title it Region.

2. Set the filter type to Single value.

3. Configure it as a parameter filter:

   • Fields: TPCH Regions (Context).region

   • Parameters: TPCH Sales (Context).region_param

If you don’t want a helper dataset, you can instead use TPCH Sales (Context).region as the field source, but the wiring is otherwise identical.

Effect

• When a viewer selects Region: EUROPE in the Region filter:

  • The widget writes “EUROPE“ into region_param for TPCH Sales (Context).

  • TPCH Sales (Context) reruns with WHERE r.r_name = ‘EUROPE’.

• All visuals built on TPCH Sales (Context) now use only European data as their starting point.

• Any additional field filters (for example, Nation, Customer, Date) operate on this already-filtered subset, just like secondary filters evaluated after a context filter.

• From the viewer’s perspective, Region behaves like a true context filter: it defines the base subset of data first, and everything else – other filters, cross-filtering, drill-through – is evaluated on top of that context.

Implementing “Only Relevant Values” with cascading filters and query-based parameters

What “Only Relevant Values” behavior means

“Only Relevant Values” behavior on a filter shrinks the list of values based on the current state of other filters and the view:

• If you select Region: ASIA, the Country filter only shows countries that actually have data in ASIA.

• As you add more filters, each filter’s domain is recomputed from the filtered dataset.

Practically, this gives you cascading filters that stay in sync with each other and with the current slice of data.

How to think about “Only Relevant Values” in AI/BI Dashboards

In AI/BI Dashboards, you get the same effect in two ways:

1. Field filters on the same dataset – AI/BI recomputes the value list based on the current filtered dataset.

2. Query-based parameters – a specialized filter widget that both populates its values from a query, and writes the selected value into a parameter used in your dataset SQL.

Pattern 1: Cascading filters with field filters

The simplest way to mimic “Only Relevant Values” is to use field filters wired to the same dataset. AI/BI Dashboards will automatically recompute each filter’s value list based on the current filtered dataset.

We’ll build a Region → Nation → Customer cascade on top of TPCH Sales (Cascading Pattern 1).

In the companion dashboard, this pattern is implemented on the “Cascading filters with field filters” page.

Step 1. Define TPCH Sales (Cascading Pattern 1)

Create a dataset TPCH Sales (Cascading Pattern 1) with the base TPCH join and a revenue metric:

SELECT
  r.r_name              AS region,
  n.n_name              AS nation,
  c.c_custkey           AS customer_id,
  c.c_name              AS customer_name,
  o.o_orderkey          AS order_id,
  o.o_orderdate         AS order_date,
  l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey;

This dataset has no parameters; all filtering will be done with field filters on top of the query results.

Step 2. Add Region, Nation, and Customer field filters

On the dashboard page where you want cascading behavior:

1. Add three field filter widgets with titles Region, Nation, and Customer.

2. Configure each widget as a page-level filter (or move them into the global filters panel if you want them to apply across pages).

3. Connect the filters to the following fields from TPCH Sales (Cascading Pattern 1):

   • Region → region

   • Nation → nation

   • Customer → customer_id

No parameters are involved here – these are pure field filters on a single dataset.

Effect

• When a viewer selects region: ASIA, the TPCH Sales (Cascading Pattern 1) dataset is filtered to ASIA for all visuals on the page.

• The Nation field filter’s value list is recomputed from that filtered dataset, so it only shows nations in ASIA.

• After the viewer chooses a nation (e.g., JAPAN), the Customer field filter shrinks to show only customers in that nation.

• From the user’s perspective, these field filters behave like filters with an “Only Relevant Values” option enabled: each dropdown shows only values that exist in the currently filtered data. Under the hood, AI/BI Dashboards are simply applying field filters on top of a single dataset and recomputing the dropdown values from the currently filtered result set.

Pattern 2: Cascading filters with query-based parameters

In Pattern 1, we used field filters only. In some cases you may want more control over how dropdown values are loaded, or you may want the same parameter to drive multiple datasets. In that case you can use query-based parameters. A query-based parameter filter widget gets its dropdown values from a field in a “choices” dataset, and writes the selected value into one or more parameters that are used in dataset SQL.

Here we’ll build a three-level cascade Region → Nation → Customer using:

• One main dataset: TPCH Sales (Cascading Pattern 2)

• Three small “value list” datasets:

  • TPCH Regions (Cascading Pattern 2)

  • TPCH Nations by Region (Cascading Pattern 2)

  • TPCH Customers by Nation (Cascading Pattern 2)

In the companion dashboard, this pattern is implemented on the “Cascading filters with query-based parameters” page.

Step 1. Define TPCH Sales (Cascading Pattern 2)

Create the TPCH Sales (Cascading Pattern 2) dataset with parameters for region, nation, and customer:

SELECT
 r.r_name              AS region,
 n.n_name              AS nation,
 c.c_custkey           AS customer_id,
 c.c_name              AS customer_name,
 o.o_orderkey          AS order_id,
 o.o_orderdate         AS order_date,
 l.l_extendedprice * (1 - l.l_discount) AS revenue
FROM samples.tpch.region   AS r
JOIN samples.tpch.nation   AS n ON n.n_regionkey = r.r_regionkey
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
JOIN samples.tpch.orders   AS o ON o.o_custkey   = c.c_custkey
JOIN samples.tpch.lineitem AS l ON l.l_orderkey  = o.o_orderkey
WHERE (:region_param   = 'All' OR r.r_name = :region_param)
  AND (:nation_param   = 'All' OR n.n_name = :nation_param)
  AND (:customer_param = 0     OR c.c_custkey  = :customer_param);

In the dataset’s Parameters panel:

• Set region_param type to String.

• Set nation_param type to String.

• Set customer_param type to Numeric / Integer (to match c_custkey).

This last bit is important: the Customer filter uses a numeric field, so the parameter must be numeric as well.

Step 2. Create helper datasets for the dropdowns

1. TPCH Regions (Cascading Pattern 2) – list of regions:

SELECT DISTINCT r_name AS region
FROM samples.tpch.region
ORDER BY region;

2. TPCH Nations by Region (Cascading Pattern 2) – nations for the selected region:

SELECT DISTINCT n.n_name AS nation
FROM samples.tpch.nation   AS n
JOIN samples.tpch.region   AS r ON n.n_regionkey = r.r_regionkey
WHERE r.r_name = :region_param
ORDER BY nation;

This dataset defines its own region_param (string) in the Data tab.

3. TPCH Customers by Nation (Cascading Pattern 2) – customers for the selected nation:

SELECT DISTINCT
  c.c_custkey AS customer_id
FROM samples.tpch.nation   AS n
JOIN samples.tpch.customer AS c ON c.c_nationkey = n.n_nationkey
WHERE n.n_name = :nation_param
ORDER BY customer_id;

This dataset defines nation_param (string). customer_id is numeric, matching customer_param in TPCH Sales (Cascading Pattern 2).

Run each dataset in the Data tab once to confirm they succeed.

Step 3. Add Region, Nation, and Customer filter widgets

On your dashboard page, add three filter widgets and wire them to fields and parameters. Configure all three widgets as page-level filters (or move them into the global filters panel if they should apply across pages).

1. Region filter widget

• Filter type: Single value

• Fields: TPCH Regions (Cascading Pattern 2).region

• Parameters:

  • TPCH Sales (Cascading Pattern 2).region_param

  • TPCH Nations by Region (Cascading Pattern 2).region_param

• Default value: All

This keeps region_param in TPCH Sales (Cascading Pattern 2) and TPCH Nations by Region (Cascading Pattern 2) in sync.

2. Nation filter widget

• Filter type: Single value

• Fields: TPCH Nations by Region (Cascading Pattern 2).nation

• Parameters:

  • TPCH Sales (Cascading Pattern 2).nation_param

  • TPCH Customers by Nation (Cascading Pattern 2).nation_param

• Default value: All

This keeps nation_param in TPCH Sales (Cascading Pattern 2) and TPCH Customers by Nation (Cascading Pattern 2) in sync.

3. Customer filter widget

• Filter type: Single value

• Fields: TPCH Customers by Nation (Cascading Pattern 2).customer_id

• Parameters: TPCH Sales (Cascading Pattern 2).customer_param

• Default value: 0

Effect

• When a viewer selects Region: AMERICA:

  • The Region widget writes “AMERICA“ into region_param in TPCH Sales (Cascading Pattern 2) and TPCH Nations by Region (Cascading Pattern 2).

  • TPCH Nations by Region (Cascading Pattern 2) reruns and returns only nations in AMERICA, so the Nation dropdown only shows those nations.

• When the viewer then selects Nation: UNITED STATES:

  • The Nation widget writes “UNITED STATES“ into nation_param in TPCH Sales (Cascading Pattern 2) and TPCH Customers by Nation (Cascading Pattern 2).

  • TPCH Customers by Nation (Cascading Pattern 2) reruns and returns only customers in UNITED STATES, so the Customer dropdown only shows those customer IDs.

• When the viewer selects a specific Customer (for example, 607):

  • The Customer widget writes 607 into customer_param in TPCH Sales (Cascading Pattern 2).

  • TPCH Sales (Cascading Pattern 2) reruns with region_param, nation_param, and customer_param applied, and all visuals built on this dataset show only orders for customer 607 in UNITED STATES / AMERICA.

• From the viewer’s perspective, Region → Nation → Customer behaves like cascading filters with “Only Relevant Values” behavior enabled. Under the hood, each dropdown is a query-based parameter filter, and the Region and Nation widgets keep parameters in multiple datasets in sync, while Customer filters the main TPCH Sales (Cascading Pattern 2) dataset down to a single customer.

Which pattern when?

Both patterns get you “Only Relevant Values”-style cascading behavior, but they shine in different situations.

Pattern 1 – Cascading field filters

Use this when:

• You’re working off one main dataset per page.

• You want the simplest authoring experience: add field filters, connect them to the dataset, done.

• “Allow All” and easy clearing of filters are important to your users.

This is the closest to what many BI tools do by default and is usually the right starting point.

Pattern 2 – Cascading query-based parameters

Use this when:

• You need parameters that drive multiple datasets.

• You want tighter control over dropdown values, including custom queries per level.

• You’re comfortable managing parameter types and wiring filters to multiple datasets.

Pattern 2 is more flexible and explicit, but also more advanced. In practice, I start with Pattern 1 for most dashboards, and reach for Pattern 2 when I need parameter-driven logic or want to reuse the same parameters across several datasets and pages.

Summary

In this post, we looked at how to carry two of the most important filter patterns from traditional BI dashboards into Databricks AI/BI Dashboards:

• Context filters become parameters in your dataset SQL, driven by parameter filter widgets. This lets you enforce “filter by Region first” semantics and shrink the working set before joins and aggregations.

• “Only Relevant Values” / cascading filters can be implemented either with simple field filters on a single dataset (Pattern 1) or with query-based parameters and helper datasets (Pattern 2) when you need more control and reusable parameters.

The companion dashboard includes all three examples: a context filter page, a field-based cascading page, and a query-based cascading page. You can import it into your workspace and adapt the patterns to your own datasets.

In future posts, I plan to cover:

• Row-level security and user-based filtering in AI/BI Dashboards

• Action-style interactions such as cross-filtering and drill-through in AI/BI Dashboards

If you’re starting a migration from an existing BI tool to Databricks AI/BI today, I recommend:

1. Identify your key context filters (Region, Business Unit, etc.) and implement them as parameters in dataset SQL.

2. Start with Pattern 1 (field filters) for cascading behavior, and only move to Pattern 2 where you truly need parameter-driven logic or shared parameters across datasets.

These two patterns alone are usually enough to make an AI/BI dashboard feel as interactive and “alive” as the dashboards your teams are used to today.