Data Today: Databricks

Databricks Unity AI Gateway hits GA: a governance guide

2026-08-10T00:00:00Z

If your team has been stitching together access to Anthropic, OpenAI, and Gemini models across a dozen notebooks with no central audit trail and no one quite sure what the monthly AI bill is, Databricks shipped something for you. Unity AI Gateway reached general availability on August 4, 2026, bringing provider routing, MCP server governance, spend caps, and usage lineage under the Unity Catalog umbrella. The GA covers the core gateway: routing traffic, enforcing access controls, and reporting cost. Some of the more interesting capabilities, including Smart Routing and agent services, remain in Beta. If you are a workspace owner who influences the bill, this is the moment to decide what goes through the gateway and what stays direct.

What does Unity AI Gateway actually do?

Unity AI Gateway is the Databricks governance solution for enterprise AI, built as a component of Unity Catalog. It sits between your applications, agents, and coding assistants and the model providers they call. Every AI request flows through the gateway, which applies your policies, records what happened, and reports back to a central dashboard.

The four core capabilities are straightforward. You control which AI services teams can use, from external LLM providers to internal MCP servers. You route and manage AI traffic across providers, reserving expensive frontier models for tasks that need them. You govern MCP servers to control access and costs. And you monitor usage, cost, access, and lineage from one place.

The gateway supports frontier models from Anthropic, OpenAI, Gemini, Kimi, and GLM through a single-query API, according to the Databricks GA announcement. That means a developer can swap the model behind a request without rewriting application code: change a config, not a codebase.

What makes this different from calling a provider API directly is that Unity Catalog's identity and permissions model extends to every AI interaction. The same RBAC you use for tables and volumes now covers AI traffic. A user who loses access to a model sees that enforced at the gateway, not just at the application layer. The gateway also provides end-to-end observability and granular cost attribution across models, providers, teams, and applications, all centralized in Unity Catalog.

How does provider routing work?

Unity AI Gateway exposes a single API endpoint that fronts multiple providers. Instead of pointing your application at a provider URL, you point it at the gateway. The gateway then forwards the request to the provider you configured.

In its GA form, routing is policy-based: you define which models are available and which teams can use them. A team building a coding assistant might get access to Claude and GPT-4, while a team running summarization gets routed to a cheaper model. You make the routing decision in configuration, and the gateway enforces it.

The more interesting piece is Smart Routing, which remains in Beta. Smart Routing dynamically routes each request to the right model based on factors like quality, cost, performance, availability, and budget. The gateway evaluates each incoming request and picks the model that best fits your configured trade-offs, reserving the most capable and expensive models for tasks that need them while routing routine work to more efficient options. Databricks says interested teams should contact their account team to join the Beta.

Here is what a basic gateway query looks like in SQL, using the Databricks ai_query function:

-- Query a model registered in Unity Catalog through the gateway
SELECT ai_query(
  'my_catalog.my_schema.my_claude_model',
  'Explain RBAC in one sentence'
);

The ai_query function sends the request through the gateway, which applies your policies, forwards it to Anthropic, and logs the token usage and cost. Swap the model name and the same function hits a different provider.

Illustrative: Unity AI Gateway capabilities split between GA (100) and Beta (50). Source: Databricks release notes, August 2026. Data Today benchmark.

The chart above shows which Unity AI Gateway capabilities shipped as GA versus Beta in the August 2026 release. The four GA capabilities are core gateway routing, provider integration via the single-query API, spend budgets and caps, and usage monitoring with Genie-powered dashboards. The four Beta capabilities are service policies, agent services, Smart Routing, and MCP connector integration.

What does it cost to run?

Here is the part that matters to anyone who signs off on the bill. Unity AI Gateway itself is a feature of Unity Catalog, not a separately priced SKU. You do not pay a gateway subscription fee. What you do pay for is the compute and token consumption that flows through it.

There are two cost surfaces to track:

Provider token costs: Every request routed through the gateway to an external provider like Anthropic or OpenAI incurs that provider's per-token pricing. The gateway records these costs in its dashboards, but the billing flows through your existing provider relationship.
Databricks compute: If you are running Databricks-hosted models via Foundation Model APIs, you pay DBUs based on the provisioned throughput. The gateway does not add a surcharge on top of provider token costs.

Cost surface	What you pay	Who bills you
External provider tokens	Per-token pricing from the provider	Anthropic, OpenAI, etc.
Databricks-hosted models	DBUs based on provisioned throughput	Databricks
Gateway infrastructure	No separate charge; included in Unity Catalog	N/A

The financial value proposition is spend control, not spend reduction. The gateway gives you proactive budgets and hard spend caps so a runaway agent cannot quietly burn through a provider quota. You set a monthly cap per team or per application, and the gateway enforces it at runtime. The out-of-the-box dashboards surface cost by model, team, and application. That attribution is the real win for a workspace owner who has been trying to explain an unpredictable monthly AI bill to finance.

How do you set it up?

Setting up Unity AI Gateway requires admin access to Unity Catalog. The high-level flow:

Register providers and models in Unity Catalog so the gateway knows what is available.
Define service policies that map teams or applications to allowed models. Note that service policies remain in Beta.
Set spend budgets per team or application using the gateway's budget configuration.
Point your applications at the gateway endpoint instead of calling providers directly.

For MCP servers, the August 6 release note confirms that all Databricks-managed MCP connectors for Genie One and Genie Code have migrated to Unity AI Gateway. If your team uses these connectors, users must reauthenticate to continue using them. A notification in Genie prompts impacted users to reconnect.

Here is how you might call the gateway from a Python application using a REST endpoint:

import requests

# Route a completion through Unity AI Gateway
response = requests.post(
    "https://<workspace>.cloud.databricks.com/api/2.0/ai-gateway/query",
    headers={"Authorization": f"Bearer {token}"},
    json={
        "model": "my_catalog.my_schema.my_claude_model",
        "messages": [
            {"role": "user", "content": "Summarize this table schema"}
        ]
    }
)

# The gateway logs the request, applies policies,
# and returns the response with token usage metadata
result = response.json()
print(result.get("usage", {}).get("total_tokens"))

The API contract is the same whether the model is hosted on Databricks or external. The gateway abstracts the provider difference. For a deeper walk through the budget and spend-cap mechanics, read our Unity AI Gateway budgets guide.

What is still in Beta and what should you wait on?

Several capabilities shipped as Beta alongside the GA core. Service policies, the engine that maps who-can-use-what at a fine-grained level, are still in Beta. You can register models and route traffic in GA, but the policy enforcement layer may not be fully stable. Agent services, including agent runtime policies and agent-level spend controls, also remain in Beta.

Smart Routing, the dynamic per-request model selection feature, is in Beta. Static routing where you pick the model is GA. MCP connector integration, while migrated to the gateway, is tagged Beta.

If your use case is routing all coding assistant traffic through one audited endpoint with a spend cap, the GA core covers you. If you want the gateway to automatically pick the cheapest model that meets a quality bar on every request, you are waiting on Smart Routing to leave Beta.

When is it the wrong choice?

Unity AI Gateway is not a model hosting platform. If your workload is a single Databricks-hosted model serving internal traffic with no external providers involved, the gateway adds a hop without adding much value. You already get lineage and access control through Unity Catalog and Model Serving.

It is also not a replacement for a full LLM observability platform. The dashboards surface cost and access, which is valuable, but if you need deep evaluation metrics like answer faithfulness, retrieval quality, or hallucination tracking, you will need a dedicated eval tool on top.

Finally, if your team has not yet adopted Unity Catalog for data governance, the gateway is not a shortcut. It depends on Unity Catalog's identity and permissions model. No Unity Catalog, no Unity AI Gateway.

What should you do this week?

If you are already on Unity Catalog and routing AI traffic to external providers, start planning the migration now.

Inventory every place your code calls an external provider API directly. Each one is a candidate for gateway routing.
Register your models in Unity Catalog so the gateway can route to them.
Set a spend cap on each team or application before you route production traffic. A cap you set before launch is a cap you control. A cap you set after an incident is a cap you explain to finance.
Reauthenticate MCP connectors if your team uses Genie One or Genie Code. The migration to the gateway requires users to reconnect.
Watch the dashboards for a week before you turn on Smart Routing. You need baseline cost data to know whether dynamic routing is actually saving you money.

The bottom line

Unity AI Gateway at GA is a governance layer, not a performance layer. It will not make your models faster or cheaper. What it does is give you one audited, policy-enforced path between your teams and the model providers they use, with spend caps and lineage that finance can actually read. If you are spending real money on external LLM providers through Databricks, the GA core is ready to use. The Smart Routing Beta is the feature to watch, but do not wait for it to start governing your AI traffic.

Sources

Databricks , August 2026 release notes
Databricks , Unity AI Gateway is Generally Available
Databricks Community , Announcement: Unity AI Gateway is Generally Available

Databricks RBAC public preview: the exclusive access shift

2026-07-27T00:00:00Z

Databricks just changed how permissions resolve in a session, and if you run governed or multi-tenant workloads, you need to understand the model before you flip the switch. Databricks RBAC, or role-based access control, entered Public Preview on July 22, 2026, and it introduces a concept the platform has never had: exclusive, role-scoped access on top of Unity Catalog. Users who assume a role get only that role's permissions for the duration of the session, not their own accumulated grants. That sounds simple, but it is a fundamental departure from the additive model Unity Catalog has used since launch.

The preview is account-level opt-in with workspace-by-workspace rollout, and it changes the permission resolution model your existing queries and pipelines already depend on.

It matters most for regulated environments where mixing data across projects, clients, or clinical trials is a compliance violation, not just a bad day at the keyboard. If you have ever answered an auditor's question with "well, technically the group grants let them see it, but nobody would actually query it," RBAC is the feature that retires that answer.

What does Databricks RBAC actually do?

The core idea is role assumption, not role assignment. In Unity Catalog today, your effective permissions are the union of every grant given to you directly plus every grant given to every group you belong to. If you are in five groups, you carry all five groups' permissions into every session, every query, every notebook. You cannot shed them. You cannot say "for this session, I only want the finance role's access."

RBAC changes that. When a user assumes a role, Databricks replaces their accumulated permissions with only that role's grants for the duration of the session. The user's own direct grants, their group memberships, their inherited access: all suspended. What remains is the role's scoped permission set, and nothing else.

Databricks calls this "exclusive access." The word does two things. It means the role's permissions are the only ones in play, and it means the user must actively choose to assume the role to reach sensitive data. They cannot stumble into it through their default identity. They cannot mix data from two roles in one session. Each role assumption is a deliberate, auditable act.

The release notes call out specific use cases: clinical trials, projects, clients. The pattern is the same in each. You have a person who needs access to multiple isolated datasets, but never simultaneously, and never in a way that lets data from one context bleed into another.

How is this different from Unity Catalog's existing permissions?

This is the question that determines whether RBAC is a net positive for your workspace or a source of headaches. Unity Catalog's existing model is additive and identity-based. You grant USE on a catalog to a group, add users to that group, and those users can query that catalog in any session, alongside every other catalog they can reach.

-- Standard Unity Catalog: grants are additive
GRANT USE ON CATALOG marketing TO analytics_team;
GRANT SELECT ON TABLE marketing.campaigns TO analytics_team;
-- User in analytics_team carries these grants in EVERY session

The model is simple, it scales for read-mostly analytics, and it has a known weakness: privilege accumulation. A user in three groups reaches everything any of those groups can touch, in every session, whether they need it or not.

Imagine a data engineer who belongs to three groups: analytics, finance, and platform admin. The chart below shows what happens to their catalog access under each model.

Illustrative: A user belonging to three groups accesses 6 catalogs through accumulated Unity Catalog grants. Under Databricks RBAC, assuming a single role exposes only that role's 2 catalogs. Source: Databricks release notes, July 2026. Data Today benchmark.

Without RBAC, the user reaches 6 catalogs through accumulated grants. With RBAC, assuming one role exposes only 2. The other 4 are invisible for that session. Not masked, not filtered behind a view. Genuinely inaccessible, because the session's permission set is the role's, not the user's.

Here is a three-way comparison of the access models now available on the platform:

Dimension	Unity Catalog groups (existing)	RBAC role assumption (new)	Service principal tokens
Permission resolution	Additive: user grants plus all group grants combine	Exclusive: only the assumed role's grants apply	Additive: all grants to the SP combine
Session scope	Every session, every tool	Duration of the assumed-role session only	Token lifetime, typically automated
Who configures it	Workspace admins via GRANT and ALTER statements	Account admin enables preview, workspace admin enables per workspace	Account admin creates SP, workspace admin grants
Best fit	Shared team analytics, broad read access	Regulated isolation, clinical trials, multi-tenant separation	CI/CD, scheduled jobs, automation
Status	GA	Public Preview	GA

The critical row is permission resolution. Every other Databricks access model is additive. RBAC is the first exclusive one. That is why it sits in a new category rather than being a refinement of existing group permissions.

How do you turn it on?

The enablement is two-step, and the order matters. An account admin turns on the preview at the account level first. Only after that can workspace admins enable RBAC in each individual workspace. There is no global on-switch that activates RBAC across every workspace in your account, which gives you a controlled rollout path: turn on the account-level preview, pick one workspace, enable it there, test, then expand.

The release notes do not specify a separate billing line item for RBAC. Based on the feature description, it appears to be a governance capability layered on top of Unity Catalog, not a separately metered service. What you pay for is the compute you run while interacting with governed data, same as today.

One thing to check before you enable anything: your existing group structure. RBAC roles are a new construct, not a rename of Unity Catalog groups. If your groups are a tangle of overlapping grants accumulated over years, you will need to clean that up before you can define meaningful roles. The role design is where the work is, not the toggle.

Also note that releases are staged. Databricks says your account might not be updated until a week or more after the July 22 release date. If you do not see the preview toggle yet, wait.

What breaks or changes when RBAC is active?

This is where you need to pay attention. When a user assumes a role, their own permissions are suspended for that session. That means:

Scheduled jobs running as a user identity may fail if the user assumes a role that lacks the job's required grants. Any pipeline that depends on accumulated group permissions to reach multiple catalogs will break when the session is role-scoped.
Notebooks that query across catalogs from different roles will fail. If your analysis joins marketing data with finance data, and those live in separate role scopes, no single role assumption gives you both. You would need two sessions and staged intermediate results.
Service principals are unaffected unless you also configure role assumption for them. RBAC applies to user sessions. Your automation, if it runs as a service principal with its own grants, continues to use the additive model.

The pattern to watch for is the same one that caught teams off guard when Databricks shipped PAT auto-scoping: a permission change that sounds like a tightening turns out to break jobs that quietly depended on the old, looser behavior. The difference here is that RBAC is opt-in and session-scoped, so it will not break anything until someone actually assumes a role. But once they do, every assumption carries the same risk.

When should you skip RBAC, and when is it essential?

Skip it if your workspace is a single-team analytics environment where everyone who can see the data is allowed to see all of the data. RBAC's value comes from isolation, and if you have nothing to isolate, you are adding ceremony for no payoff. The enablement overhead, the role design, the session management: all cost time that buys you nothing.

Skip it too if your access control problem is "some people should not see some columns." That is a masking problem, and Unity Catalog already handles it with column-level masking and row-level filters. RBAC is about catalog- and table-level isolation between distinct contexts, not about hiding sensitive fields within a shared dataset.

Use it if you operate in any of these patterns:

Clinical trials or life sciences, where an analyst working on Trial A must not have even accidental visibility into Trial B's data, and where an auditor needs proof that the separation was enforced at the session level.
Multi-tenant SaaS or consulting, where each client's data lives in the same workspace but must never cross-contaminate, and where a single engineer rotates across clients.
Financial services with conflict-of-interest walls, where an analyst covering one sector must not access data about companies in a conflicted sector, and where the penalty for accidental access is regulatory.

In each case, the value is the same: the user's default identity cannot reach the sensitive data. They must assume a role, which creates an auditable event, and the role's permissions are the only ones in play.

What should you plan for during the preview window?

Three things, in order.

First, map your current permission graph before you touch anything. Query your Unity Catalog grants, list every group membership for the users who would assume roles, and identify where accumulated permissions are load-bearing for existing pipelines. The information schema in Unity Catalog can give you this.

Second, design your roles around isolation boundaries, not job functions. A role called "marketing_analyst" is too broad. A role called "client_acme_readonly" or "trial_42_analyst" is specific enough that the exclusive access model gives you real isolation. The granularity of your roles determines the granularity of your compliance.

Third, test with one workspace, one team, and one non-production role before you expand. The preview is opt-in and per-workspace for a reason. Use that. Enable it in a sandbox workspace, create a role, have a user assume it, and verify that the permission resolution matches your expectations. Check that queries to out-of-scope catalogs fail with permission errors, not silent empty results.

Watch for changes to the configuration surface. Public Preview features can shift their API, their UI, and their behavioral semantics between now and GA. Databricks also notes that releases are staged, so your account may not see the feature until a week or more after the July 22 release date. Anything you automate around RBAC today may need adjustment before general availability.

The bottom line on exclusive access

RBAC is the first feature on Databricks that lets you say "this session can only see this data, and nothing else." For teams operating under regulatory constraints, that is a capability they have been working around with separate workspaces, separate accounts, and awkward process controls. Now it is a session-level property. The trade-off is that exclusive access breaks the pattern every other part of the platform follows, and it requires deliberate role design and testing to avoid breaking pipelines that depend on accumulated permissions. Enable it where isolation matters. Leave it off where it does not.

Sources

Databricks managed Iceberg materialized views, explained

2026-07-13T00:00:00Z

Databricks has spent years building materialized views that precompute and cache query results inside its own SQL warehouses, making dashboards and BI faster. The catch was always that those MVs spoke Delta, not Iceberg, so if your platform mix includes Trino, Spark on Kubernetes, or anything else that reads open table formats, your carefully maintained MV was invisible to it. That changes with managed Iceberg materialized views, which entered Public Preview on July 7, 2026 and let you create a materialized view that external Iceberg readers can query directly. For teams running mixed Databricks and open-source stacks, this is the feature that reframes the lock-in conversation.

The feature writes MV results to Apache Iceberg instead of Delta Lake, which means any engine with an Iceberg reader can open the same physical files your Databricks SQL warehouse produced.

What are managed Iceberg materialized views?

A materialized view is a precomputed result set stored as a table and refreshed on a schedule or trigger. Databricks SQL has supported them for a while, letting you accelerate expensive aggregations, joins, and transformations by computing once and reading many times.

The new twist is the storage format. Standard Databricks materialized views are written to Delta Lake tables. Managed Iceberg materialized views write to Apache Iceberg instead. Any engine with an Iceberg reader, including Trino, Apache Spark, and DuckDB, can query the same physical table your Databricks SQL warehouse produced. You get the precompute benefit inside Databricks and the portability benefit outside it.

The feature is in Public Preview. To enable it, you need to contact your Databricks account team; there is no self-service toggle in the workspace UI. Databricks documents the setup under Create a materialized view compatible with external Iceberg readers in the July 2026 release notes.

The distinction matters because it changes how you think about MVs on this platform. If your analytics estate is purely Databricks, standard Delta MVs are simpler and fully supported. If a second engine needs the same precomputed results, Iceberg MVs eliminate the need to maintain a parallel copy or export pipeline.

How do you configure them?

Since this is Public Preview and account-team-gated, the first step is contacting your Databricks representative to enable the feature for your workspace. Once enabled, the creation path mirrors standard materialized views but directs the output to Iceberg-format storage.

A typical materialized view definition follows the standard Databricks SQL pattern:

CREATE MATERIALIZED VIEW catalog.schema.daily_revenue
COMMENT 'Daily revenue by region, Iceberg-readable'
AS
SELECT region, order_date, SUM(amount) AS revenue
FROM catalog.schema.orders
GROUP BY region, order_date;

The exact property or catalog configuration that targets Iceberg rather than Delta is described in the Databricks documentation linked above. The refresh happens on a schedule or on demand, the same as a standard MV. What changes is that the resulting table is queryable by any Iceberg-compatible engine that can reach your catalog storage.

Refreshes can also take advantage of REPLACE WHERE flows, which became generally available on July 7, 2026. REPLACE WHERE lets you replace only the rows matching a predicate on each refresh without reprocessing the full table history. For a daily-revenue MV partitioned by date, that means you can recompute just yesterday's partition instead of rebuilding the entire view. This applies to pipelines and standalone streaming tables as well.

What does it cost you?

Materialized view refreshes consume compute, and Iceberg MVs are no exception. The cost story has two relevant updates from the July 2026 release.

First, materialized view and streaming table refreshes in Databricks SQL now inherit custom tags from the enclosing SQL warehouse environment, as of July 8, 2026. These tags propagate to system.billing.usage, so you can attribute refresh costs back to the source SQL warehouse. If you have been unable to figure out which warehouse is paying for which MV refresh, this closes that gap for both Delta and Iceberg MVs alike.

Second, materialized view and streaming table refreshes orchestrated through a SQL task in a Lakeflow job now respect the job's performance-optimized setting, as of July 6, 2026. When you turn that setting off, the refresh uses standard performance mode, which consumes fewer DBUs than the performance-optimized default. For Iceberg MVs that refresh on a schedule, choosing standard mode when latency is not critical is a straightforward way to cut the refresh bill.

Here is how the two modes compare:

Setting	Performance mode	Standard mode
DBU consumption	Higher	Lower
Refresh speed	Faster	Slower
When to use	SLA-sensitive refreshes	Overnight or batch refreshes
Configurable via	Lakeflow job SQL task setting	Same setting, toggled off

The cost attribution tags work regardless of which performance mode you choose. You can read more about managing compute spend in our Databricks Lakeflow Designer cost guide, which covers the same tag-based attribution pattern for pipeline workloads.

The chart below shows the illustrative difference in DBU consumption between performance and standard mode for scheduled MV refreshes at three data volumes. Standard mode uses roughly 30 percent fewer DBUs in this example, though the exact ratio depends on your query complexity, data volume, and warehouse size. Treat the numbers as directional, not as a Databricks-published rate.

Illustrative: DBU consumption for a typical scheduled materialized view refresh under performance-optimized and standard modes, showing standard mode at 56, 84, and 140 DBUs versus performance mode at 80, 120, and 200 DBUs for small, medium, and large MVs respectively. Source: Data Today illustrative benchmark.

When are they the right choice?

The use case is narrow but compelling. You should consider managed Iceberg materialized views when a second compute engine outside Databricks needs to query your precomputed results, and you are already paying to maintain a parallel copy or export pipeline that duplicates MV logic in another system. Trino and Spark on Kubernetes are the most common examples.

Another fit is when your governance or contractual requirements mandate open table formats for downstream consumption, or when you want to evaluate a migration away from Databricks for read workloads without rewriting your transformation logic.

If none of those apply, standard Delta materialized views remain the simpler choice. They are GA, fully supported, and do not require an account-team conversation to enable. The Iceberg format adds a portability tax in the form of preview-stage limitations and format overhead that is not worth paying if only Databricks will ever read the table.

The feature also fits into the broader Databricks Data Intelligence Platform story of meeting open-format demand without forcing teams off the platform. Unity Catalog governs both Delta and Iceberg tables, so permissions, lineage, and auditing apply consistently across formats.

What are the limits in Public Preview?

Public Preview means the feature is real but not finished. Several constraints are worth flagging before you build a production pipeline around it.

Account-team enablement required. There is no self-service toggle. You file a request and wait.
Staged rollout. Databricks stages releases across accounts, so your workspace may not have the feature for a week or more after the July 7 date, and you should expect rough edges in edge cases.
Refresh semantics. Incremental refresh and REPLACE WHERE support depend on the underlying MV's partitioning and the query pattern. Not every MV can be refreshed incrementally; some require full recomputation.
External reader compatibility. Iceberg is a standard, but engine versions matter. Verify that your Trino or Spark version supports the Iceberg table version and features Databricks writes. Format divergence between writer and reader is a real risk in mixed stacks.

You should also confirm how Databricks bills the Iceberg-format MV refresh versus the Delta equivalent. The release notes do not call out a separate pricing tier, which suggests the same DBU model applies, but Public Preview pricing can change. Check the Databricks pricing page for current rates before committing.

What should you do next?

If you have a mixed-engine stack, the bet to make is small and deliberate. Pick one MV that a non-Databricks engine already needs, enable the feature, and migrate that single view to Iceberg format. Measure the refresh cost, the read latency from both engines, and any format-compatibility issues. That tells you whether to expand.

The bet to skip is a wholesale migration of all MVs to Iceberg format on day one. Public Preview is for validation, not for replatforming your entire semantic layer. If your team is purely on Databricks, there is no reason to move yet.

Watch for the GA announcement, which will likely remove the account-team gate and add self-service enablement. Also watch for any changes to how Unity Catalog exposes Iceberg MV metadata, since lineage and access audit across formats is where governance complexity tends to hide.

The lock-in question has a new answer

The strongest argument against building your semantic layer inside Databricks has always been that the output is Delta and the rest of the world is moving toward Iceberg. Managed Iceberg materialized views do not eliminate lock-in, but they give you a credible exit ramp for read workloads, and they let you serve two engines from one precomputed table. For a data engineer who has been maintaining duplicate pipelines just to feed Trino, that is the detail worth testing in Public Preview.

Sources

Databricks Lakehouse//RT: sub-second SQL reads in beta

2026-07-06T00:00:00Z

If you have ever pointed a dashboard or a customer-facing app at a Databricks SQL warehouse and watched the spinner tick past one second, Lakehouse Real-Time is the feature you have been waiting for. Now in Beta as of June 30, 2026, Lakehouse//RT is a new serverless SQL warehouse type built for low-latency, high-concurrency workloads against Unity Catalog tables. The promise is sub-second SELECT queries for hundreds to thousands of concurrent users.

Lakehouse//RT is a purpose-built read-only serving engine. Databricks positioned it explicitly for serving analytical data to custom applications, operational analytics, and BI dashboards in its June 2026 release notes. The feature is in active Beta, meaning performance characteristics and the supported feature set will change before general availability.

What is Lakehouse//RT and how is it different from a serverless SQL warehouse?

Lakehouse//RT is a distinct warehouse type, not a configuration flag on an existing SQL warehouse. You create it from the same SQL Warehouses UI, but it shows up as "Real-Time" alongside "Serverless" and "Pro." You cannot upgrade an existing SQL warehouse to Lakehouse//RT, and you cannot downgrade a Lakehouse//RT warehouse back. It is its own animal.

The engine is optimized for point lookups and short aggregations against well-clustered Unity Catalog managed tables. It supports Delta Lake and Apache Iceberg managed tables, materialized views, streaming tables, and metric views. For best performance, Databricks recommends managed tables with predictive optimization and liquid clustering enabled, because the engine depends on good data layout to hit sub-second latency.

Dimension	Lakehouse//RT	Serverless SQL	Pro SQL
Target latency	Sub-second	Seconds	Seconds to minutes
Concurrency target	Hundreds to thousands	Tens to hundreds	Tens
Query types	SELECT only	Full DDL, DML, SELECT	Full DDL, DML, SELECT
Connectivity	Statement Execution API only	Thrift and Statement Execution API	Thrift and Statement Execution API
Table support	UC managed, MVs, streaming tables, metric views	All Unity Catalog tables	All Unity Catalog tables
Billing SKU	Lakehouse_Serverless	Serverless_SQL	Classic_SQL

The connectivity distinction is the one most likely to bite you. Lakehouse//RT rejects any connection that does not use the Statement Execution API. If your BI tool or driver connects via the legacy Thrift protocol, it receives a 501 error. That rules out tools that have not adopted the Statement Execution API, which includes some older JDBC drivers and legacy BI connectors. You need to verify your client stack before planning a migration.

The chart below shows how Databricks positions the three warehouse types against each other on the two dimensions that matter most for serving workloads.

Illustrative: Target latency and concurrency by Databricks SQL warehouse type. Lakehouse//RT is positioned for sub-second reads and thousands of concurrent users. Source: Databricks documentation.

Lakehouse//RT sits in a different quadrant from both serverless SQL and Pro SQL: sub-second target latency with thousands of concurrent users, compared to seconds and tens for the other two.

How do you enable and create a Lakehouse//RT warehouse?

The feature is gated behind a workspace-level preview. A workspace admin enables it by navigating to the workspace menu in the top-right corner, selecting Previews, searching for "Lakehouse RT," and toggling it on. Until that toggle is flipped, the Real-Time warehouse type does not appear in the creation flow.

Once enabled, creating a warehouse is straightforward:

Go to Compute, then SQL Warehouses, then Create SQL Warehouse.
Select Real-Time as the warehouse type.
Choose a size: Small, Medium, Large, or X-Large.
Name the warehouse and click Create.

Permissions work the same as any SQL warehouse. Grant Can use, Can monitor, or Can manage to users and groups.

For connecting, the Statement Execution API is your only path. A basic request looks like:

curl -X POST \
  https://<workspace-url>/api/2.0/sql/statements \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "warehouse_id": "<rt-warehouse-id>",
    "statement": "SELECT count(*) FROM catalog.schema.table WHERE dt = '\''2026-06-30'\''",
    "format": "ARRAY"
  }'

No Thrift, no JDBC URL with the old protocol. If your application or BI tool cannot speak the Statement Execution API, Lakehouse//RT is invisible to it.

What can it actually run, and what breaks?

Lakehouse//RT runs ANSI-compliant SELECT queries. That is the full scope. Write commands, DDL, DML, and ETL operations are all unsupported. If a query uses an unsupported feature, the engine returns an error naming the feature and telling you to use a serverless SQL warehouse instead.

Several specific capabilities are off the table:

The GEOGRAPHY and GEOMETRY data types are not supported.
AI functions are not supported.
Python UDFs are not supported.
Spatial SQL functions and XPath and XML functions are not supported.

On the governance side, the picture is better. Lakehouse//RT does support attribute-based access control, including row-level security and column masking. Your serving queries respect the same Unity Catalog policies as your batch workloads, without duplicating policy logic in the application layer.

The practical implication: if your dashboard or application query calls a Python UDF to enrich results, or uses an AI function to classify a row on the fly, that query will fail on Lakehouse//RT. You need to move that logic upstream into a materialized view or a streaming table so the computed result is already on disk when the serving query runs.

You can monitor Lakehouse//RT queries the same way you monitor any SQL warehouse query. Queries appear in the query history UI and the query history system table. You can open a Lakehouse//RT query in the query history UI to view its profile. The monitoring page for each warehouse shows query throughput, queued queries, and query history.

How does the billing work?

Lakehouse//RT is serverless, so you pay for compute on demand with no idle clusters to shut down. Usage appears in the billing system tables under the SKU name Lakehouse_Serverless. That is the same billing surface you already use to track serverless SQL spend, but with a distinct SKU so you can separate Lakehouse//RT costs from your standard serverless warehouse costs.

Databricks has not published specific per-DBU pricing for Lakehouse//RT in the Beta documentation. The four sizes, Small through X-Large, follow the same naming convention as other SQL warehouses, but the Beta status means pricing and performance characteristics are subject to change. Treat any cost estimates as provisional until the feature reaches GA.

The cost question really comes down to concurrency. If you are currently running a Pro SQL warehouse sized for peak concurrency and it sits idle for 16 hours a day, Lakehouse//RT could lower your bill by serving the same traffic from a serverless pool that scales to zero. If your workload is a handful of analysts running ad hoc queries, you will not see a meaningful difference because your existing serverless warehouse already handles that pattern.

For more on how the serving layer connects to the rest of the platform, see our Databricks Data Intelligence Platform overview, which covers how Unity Catalog, Mosaic AI, and the compute layers fit together.

When is Lakehouse//RT the wrong choice?

If your workload is write-heavy, uses ETL, or depends on UDFs and AI functions, Lakehouse//RT is the wrong tool. It is a serving engine. Treating it as a general-purpose warehouse will produce errors and frustration.

It is also wrong if your client stack relies on Thrift. The Statement Execution API restriction is a hard limit. Some BI tools and application frameworks have not yet adopted it, and until they do, they cannot connect. Before you plan a migration, inventory every client that touches your SQL warehouse and confirm each one can use the Statement Execution API.

Finally, it is wrong for exploratory analytics. If your users run complex multi-table joins with subqueries and window functions in an ad hoc fashion, the engine is not optimized for that. The best-practices guidance from Databricks is explicit: validate that your queries run in a few seconds on a serverless SQL warehouse first, then move them to Lakehouse//RT. If a query takes 10 seconds on serverless, it will not take 500 milliseconds on Lakehouse//RT.

What should you do before moving a workload over?

Start with the queries, not the warehouse. Pick the dashboard or application endpoint you want to accelerate and audit its SQL. Confirm every statement is a pure SELECT with no unsupported functions or data types. Run each query on a serverless SQL warehouse and verify it completes in a few seconds.

Then look at your data layout. The single biggest factor in hitting sub-second latency is using Unity Catalog managed tables with predictive optimization and liquid clustering. If your tables are external, unclustered, or large with no partition pruning, the engine cannot help you. Materialized views that pre-aggregate complex joins are your friend here.

A checklist before you flip the switch:

Enable the Lakehouse RT preview in your workspace under Previews.
Confirm your client uses the Statement Execution API, not Thrift.
Validate target queries run in seconds on serverless SQL.
Move computed columns and UDF logic into materialized views or streaming tables.
Ensure tables are managed with liquid clustering and predictive optimization.
Filter early with WHERE clauses, select only needed columns, and lean on aggregations.
Check that row-level security and column masking policies carry over correctly.

Databricks also recommends checking that your workloads are ANSI compliant and avoiding the unsupported statements, functions, and data types listed in the Limitations section of the Lakehouse Real-Time documentation. When a query hits an unsupported feature, the error message names it.

The real test is your p99

Lakehouse//RT is a bet that a purpose-built read-only serving engine can beat a general-purpose SQL warehouse on latency and concurrency for the specific workload of serving analytical data to applications. The Beta restrictions are real: SELECT only, Statement Execution API only, no UDFs, no AI functions, no spatial types. But if you have a dashboard that thousands of users hit every minute, or a customer-facing app that needs fresh lakehouse data in under a second, this is the first Databricks warehouse type built for that job. Enable the preview, point your fastest dashboard at it, and measure your p99 latency at peak concurrency against what your current serverless warehouse delivers.

Sources

Databricks - Lakehouse Real-Time documentation
Databricks - June 2026 release notes
Databricks - Real-Time Lakehouse product page

Unity AI Gateway budgets: the spend guardrail guide

2026-06-29T00:00:00Z

AI cost control used to be a spreadsheet problem. Then Genie, coding agents, ai_query, and model endpoints put LLM spend directly in the path of normal work. That is a better product experience and a worse surprise-invoice experience.

Unity AI Gateway budgets are Databricks account-level controls for monthly AI spend. In July 2026, Databricks says Unity AI Gateway budgets become generally available across AWS, Azure, and Google Cloud, while Unity AI Gateway itself remains in Beta. The key fact for data teams: admins can set shared and per-user spending thresholds for requests managed through Unity AI Gateway and Genie, with actions such as email alerting or usage blocking.

That split matters. A data engineer does not need another dashboard that explains yesterday's bill. You need a control plane that stops one overexcited analyst, agent loop, or internal app from turning a demo into an account-wide cost incident.

What exactly becomes generally available in July 2026?

Databricks is making the budget feature generally available, not the whole gateway surface. The release note is explicit: Unity AI Gateway budgets roll out for all accounts on AWS, Azure, and Google Cloud in July 2026, but Unity AI Gateway itself remains in Beta and accounts that receive budgets are not automatically enrolled in Unity AI Gateway.

That is the first trap to avoid in your rollout plan. Budget availability does not mean all gateway traffic controls are automatically active in every workspace. For a gateway-specific budget, Databricks still lists three requirements in the current docs: Unity AI Gateway enabled for the account, the billable usage system table enabled, and the Unity AI Gateway Budget Public Preview enabled in the account console before GA arrives. The Unity AI Gateway budget docs also say endpoint tags propagate into system.billing.usage.custom_tags, which is what makes team, project, and cost center scoping practical.

Here is the useful mental model:

Control	What it governs	Confirmed scope	Where you manage or inspect it
Account budget	Total Databricks usage or filtered usage	Monthly USD list-price spend	Account console Usage tab or Budgets API
Unity AI Gateway budget	Gateway endpoint spend	Pay-Per-Token and `ai_query` inference	Account console, budget details, billing tables
Genie budget	Genie product LLM spend	Genie, Genie Spaces, Genie Code, and Genie One through `databricks-product: genie`	Account console with shared, per-user, and override controls

The feature is narrow in the right way. It does not try to become a model router, prompt firewall, FinOps warehouse, and procurement system in one gulp. It takes billing records, filters them, and lets an account admin decide when a line gets noisy or blocked.

How do Unity AI Gateway budgets actually measure spend?

Budgets are measured in US dollars using Databricks list prices, including platform add-ons. The account budget documentation says the spent amount does not factor in negotiated discounts or billing credits, so the number you see is a guardrail number, not necessarily the final invoice number.

That is the right trade for a runtime control. If you wait for negotiated net cost, you get accuracy after the blast radius. List price gives you a conservative tripwire while the workload is still live.

For Unity AI Gateway specifically, Databricks says budget scope can use the Unity AI Gateway resource type and endpoint tags such as team, project, or cost_center. The gateway budget page says Unity AI Gateway budgets currently track Pay-Per-Token, also called PAYGO, and ai_query batch inference, while provisioned throughput and external-model inference are not currently tracked.

That last clause should shape your design. If your AI estate includes provisioned throughput endpoints or direct external provider billing, a Unity AI Gateway budget will give you a partial control plane. Useful, yes. Complete, no.

The billing table is where engineering teams should verify what the console tells them. The cost observability docs say Unity AI Gateway enriches MODEL_SERVING records in system.billing.usage with fields such as usage_metadata.ai_gateway_endpoint_name, usage_metadata.ai_gateway_destination_model, identity_metadata.run_by, and custom_tags.

A practical query for the platform team:

SELECT
  custom_tags['team'] AS team,
  usage_metadata.ai_gateway_endpoint_name AS endpoint_name,
  identity_metadata.run_by AS run_by,
  SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE billing_origin_product = 'MODEL_SERVING'
  AND usage_metadata.ai_gateway_endpoint_name IS NOT NULL
  AND usage_unit = 'DBU'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY team, endpoint_name, run_by
ORDER BY dbus DESC;

Notice the billing shape. You can hold teams accountable only if they send traffic through endpoints with tags that survive into custom_tags. If the endpoint is untagged, your budget can still catch global spend, but your allocation story gets mushy fast.

How should you configure a Genie budget without blocking the wrong people?

Genie is the reason this feature deserves attention from data engineers, not just account admins. Databricks says Genie products move to pay-as-you-go pricing on July 6, 2026, with each identified user receiving 150 DBUs of free LLM usage every month, equal to about $10.50 in US East and roughly 80 to 100 Genie questions or 20 to 30 Genie Code coding sessions.

The free tier is generous enough for casual use and too small to ignore for broad rollout. A 300-person analytics org can burn through the free layer unevenly: 250 people ask a few questions, 20 power users live in Genie Code, and 5 people accidentally discover an expensive workflow. A seat license would hide that shape. Pay-as-you-go exposes it.

For Genie, Databricks documents a specific budget pattern. The Genie budget docs say you create the budget with resource type Unity AI Gateway and the resource tag databricks-product: genie, and they warn that adding other resource tags to a Genie budget is not supported and prevents the budget from tracking Genie usage.

Use that rule exactly:

Resource type: Unity AI Gateway
Resource tag key: databricks-product
Resource tag value: genie

Then layer the controls:

Set a shared monthly threshold for the workspace or account segment.
Set a per-user threshold for normal users.
Add overrides for trusted groups such as genie-code or power-users.
Use alerts first, then block usage only where the failure mode is acceptable.

Databricks gives a concrete example configuration in its docs: $5,000 shared, $100 per user, $200 for genie-code, and $300 for power-users. The chart below shows that shape as a control design, not a recommendation for every account.

Illustrative: Databricks Genie budget documentation shows an example with a $5,000 shared threshold, $100 per-user threshold, $200 genie-code override, and $300 power-users override. Source: Databricks Genie budgets documentation. Data Today benchmark.

The override logic has teeth. The Genie docs say that within one budget, a user in multiple groups inherits the most permissive threshold, so a user in both genie-code and power-users gets $300 in the documented example. Across multiple Genie budgets, the most restrictive limit applies, so a $100 limit in one budget beats a $200 limit in another.

That is sane, but it also means budget sprawl can create confusing support tickets. If you let every workspace admin create their own Genie budget, a blocked user may be blocked by a different budget than the one your team is looking at. Keep ownership centralized until your naming and group model are boring.

Where does blocking help, and where can it hurt?

Blocking is the feature that changes this from observability to control. The budget docs say thresholds can trigger email notifications, and for Unity AI Gateway budgets the known limitations include a small amount of spend beyond a blocking threshold because active requests are not interrupted and enforcement has a brief delay.

That caveat matters less for a human asking Genie questions. It matters more for an agentic workflow that can generate a lot of calls in a tight loop. If you expose Genie or gateway-backed functionality inside an internal portal, usage blocking should be part of the launch checklist, not a follow-up after finance asks why Tuesday looks weird.

A sane default policy looks like this:

Workload	Budget action	Why
Exploratory Genie use	Alert at shared threshold, block at per-user threshold	Preserve broad access while stopping individual runaway spend
Executive dashboard backed by Genie	Alert first, avoid hard block unless there is a fallback	A blocked dashboard becomes a business incident
Agent or app calling gateway endpoints	Block at project or service-principal boundary	Loops and retries can outspend humans quickly
External-model path billed by provider	Do not rely on Unity AI Gateway budget alone	Databricks says external-model inference is not currently tracked by these budgets

The right policy is also a product decision. If Genie is replacing ad hoc analyst requests, blocking at $30 may be penny-wise and queue-building. If Genie is embedded in a public-facing workflow with weak throttling, a hard stop may save the month.

This is where the broader Databricks Data Intelligence Platform architecture matters. Cost control belongs beside identity, Unity Catalog permissions, and workload ownership. Treat AI spend as a governed workload dimension, not a chat feature tucked into the corner of BI.

Can you automate budgets, or is this console-only for now?

There is a public Budgets API, but Databricks documentation is still cleaner for basic monthly budgets than for every Genie-specific UI control. The Budgets API reference exposes POST /api/2.1/accounts/{account_id}/budgets, requires the billing API scope, and shows fields such as display_name, filter, alert_configurations, quantity_threshold, quantity_type, time_period, and trigger_type.

A minimal account budget creation payload looks like this:

curl -X POST \
  https://accounts.cloud.databricks.com/api/2.1/accounts/$ACCOUNT_ID/budgets \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "budget": {
      "display_name": "ai-gateway-ml-platform",
      "filter": {
        "tags": [{
          "key": "team",
          "value": {"operator": "IN", "values": ["ml-platform"]}
        }]
      },
      "alert_configurations": [{
        "time_period": "MONTH",
        "trigger_type": "CUMULATIVE_SPENDING_EXCEEDED",
        "quantity_type": "LIST_PRICE_DOLLARS_USD",
        "quantity_threshold": "1000",
        "action_configurations": [{
          "action_type": "EMAIL_NOTIFICATION",
          "target": "ml-platform-oncall@example.com"
        }]
      }]
    }
  }'

Do not overread that snippet. It demonstrates the public budget API shape and tag filtering. The current Databricks Genie budget page documents shared thresholds, per-user thresholds, blocking, and overrides in the account console flow. If you need those exact Genie controls as code, verify the live API and provider support in your account before promising GitOps parity.

There is also a CLI surface. The Databricks CLI account budgets command group can create, list, get, update, and delete budget configurations, with JSON passed through --json. That makes it useful for inventory and drift checks even if your first rollout uses the UI for the per-user pieces.

What would I do before turning this on for a real team?

Start with attribution, not thresholds. A $5,000 budget attached to a bad tag model is a smoke alarm in the wrong building.

For Unity AI Gateway endpoints, standardize endpoint tags before you standardize alert values. At minimum, require team, env, and cost_center on every endpoint that can generate billable traffic. Databricks says only AI Gateway endpoint tags are propagated to custom_tags for budget scoping, and budgets do not support request tags, so request-level tags are useful for dashboards but not for budget filters.

For Genie, decide whether your cost boundary is workspace, user group, or individual user. The docs say Genie budgets are shared across Genie, Genie Spaces, and Genie Code under the single databricks-product: genie resource tag, so separate product budgets need to be modeled through groups rather than separate product tags.

Then run this checklist:

Confirm system.billing.usage is enabled for the account.
Create one account-level AI budget in alert-only mode for the first month.
Create a Genie budget scoped with databricks-product: genie and no extra resource tags.
Set per-user thresholds for normal users and higher overrides for named groups.
Query system.billing.usage weekly and compare the table to the budget details page.
Add blocking only where the user experience is understood.

The strongest use case is controlled self-service. Give analysts and engineers Genie access without turning every prompt into a procurement conversation. Give app teams gateway endpoints without making finance reverse engineer MODEL_SERVING records after the fact.

The weakest use case is pretending this controls all AI spend. It does not cover every serving mode today, and it does not replace provider-side budgets for external models. The honest architecture is layered: Databricks budgets for Databricks-billed gateway and Genie usage, provider budgets for provider-billed calls, and SQL checks over billing tables for audit.

The useful guardrail is the one users can hit

The best thing about Unity AI Gateway budgets is that they are close to the work. Genie questions, ai_query inference, and gateway endpoints are where AI cost enters a Databricks account. Putting shared and per-user thresholds there is more useful than explaining the invoice later with a pie chart.

The risk is false comfort. GA budgets make the control plane easier to adopt, but they do not make every AI path governed by default. If your team exposes Genie broadly in July 2026, budget it on day one. If your agents call models outside the tracked paths, budget those too.

AI spend does not need a war room. It needs boring limits, good tags, and one person willing to say that a demo gets $100 before it gets $10,000.

Sources

Databricks PAT auto-scoping is a quiet job breaker

2026-06-22T00:00:00Z

Databricks PAT auto-scoping sounds like housekeeping until a build agent, ingestion script, or BI connector loses an API it only calls on quarter close.

Databricks made token auto-scoping generally available on June 18, 2026, and the useful fact is simple: Databricks observes API usage for 30 days, then narrows a personal access token to the scopes it inferred from that usage. The June 2026 release notes say the feature is GA, enabled by default for new long-lived PATs of 30 days or longer, and applied to existing tokens with all APIs access based on historical usage.

That is good security. It is also a production change aimed directly at lazy automation.

A personal access token, or PAT, authenticates workspace level calls to Databricks APIs. Databricks still supports PATs, but its own authentication docs recommend OAuth where possible for user account authentication. Auto-scoping is the bridge for teams that still have PATs spread through GitHub Actions, Airflow, dbt jobs, Tableau extracts, shell scripts, and that one Jenkins worker nobody wants to touch.

If you influence the Databricks bill, your cost risk is indirect. This feature does not ask you to pick a warehouse size or a Databricks Runtime. The meter shows up when a token loses a scope, jobs retry, engineers investigate, SLAs slip, and someone reopens the all APIs barn door.

What exactly changed for Databricks PATs?

Before this change, many PATs behaved like broad workspace keys unless a user or admin explicitly chose API scopes. Databricks now has a default path toward least privilege for long-lived tokens.

The PAT documentation says scoped personal access tokens restrict permissions to specific API operations, with examples such as sql, unity-catalog, and scim. That same page says auto-scoping narrows a token to only the APIs it actively uses after a 30 day observation period.

The release note matters because it changes the default posture. New long-lived tokens, meaning 30 days or longer, are now in the auto-scoping path by default. Existing all APIs tokens can also be narrowed based on historical usage.

Here is the operational shape.

Token situation	Databricks behavior	What you should check
New PAT with lifetime of 30 days or longer	Auto-scoping is on by default during creation	Confirm the first 30 days include every API the automation will need
Existing PAT with all APIs access	Auto-scoping can apply based on historical API usage	Find low frequency API calls before Databricks infers the token is narrower than reality
Manually scoped PAT	Manual scopes define access	Keep it manual if the token has known, stable duties
Token with `authentication` scope	Can create new tokens with any scope	Treat it like a privileged credential, not a normal job secret

The dangerous row is the second one. Existing all APIs tokens often belong to older automation. Those tokens are exactly where unusual calls hide: a monthly jobs update, a quarterly unity-catalog permission change, an emergency clusters restart, or a one time workspace export.

The default creation flow also changed. The PAT page now includes an autoscope_enabled field in the Token API example:

curl -X POST https://<databricks-instance>/api/2.0/token/create \
  -H "Authorization: Bearer <existing-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "lifetime_seconds": 2592000,
    "scopes": ["sql"],
    "autoscope_enabled": true
  }'

That 2592000 value is 30 days in seconds. Use it as a mental boundary: below that lifetime, Databricks is pushing you toward shorter lived credentials; at that lifetime or above, Databricks is also pushing you toward observed least privilege.

How does auto-scoping decide what a token can do?

Databricks says it observes API usage over 30 days and then applies inferred scopes to the token. During the observation window, the all-apis scope is applied temporarily, so existing automation can keep running while Databricks watches which endpoints the token actually calls.

The lifecycle numbers are easy to remember, and the chart below is the one to put in your admin runbook: 30 days of observation, a 7 day reminder before enforcement, 90 days before unused PATs are automatically revoked, and 730 days as the default maximum lifetime for new tokens unless you configure a shorter maximum.

Databricks PAT controls from the June 2026 release notes and authentication documentation: 30 days of auto-scoping observation, a 7 day reminder before enforcement, 90 day unused token revocation, and a 730 day default maximum lifetime. Source: Databricks documentation and June 2026 release notes. Data Today benchmark.

Databricks also says it sends a reminder email 7 days before enforcement and another email when scopes are applied. That helps, but it does not replace an audit. Service accounts, service principals, and human users have different inbox realities, and the person who receives the warning may have no idea what job owns the token.

The inference model is endpoint driven. Databricks publishes an API scopes reference that maps scopes to API families, including jobs with 40 operations, sql with 53 operations, unity-catalog with 123 operations, clusters with 36 operations, and mlflow with 89 operations. The point is practical: a token that successfully queries Databricks SQL with sql still cannot necessarily manage a Lakeflow job or update Unity Catalog metadata.

This is where builders get bitten. Workloads rarely fail on their main path. They fail on a repair path.

A nightly script might use sql every evening and jobs only when it rotates a schedule. A Lakeflow related automation might run normal pipeline calls through one scope and need workspace or secrets during deployment. A model promotion job might call MLflow every release and Unity Catalog only when registering a new governed model.

For a broader platform view, this fits the same theme as our guide to the Databricks Data Intelligence Platform for builders: Databricks keeps moving governance into default platform behavior, which is great until your automation relied on informal privilege.

How do you audit tokens before access tightens?

Start with the token inventory, not the codebase. Code lies by omission. Token metadata tells you what exists.

Databricks says workspace admins can use the CLI or Token Management API to monitor and revoke existing workspace tokens, and its admin docs show databricks token-management list as the basic command. The same admin page says account admins can filter a token report by owner, workspace, created date, expiration date, and last used date.

databricks token-management list

If you are working inside a notebook or admin utility, Databricks shows the SDK pattern for loading token metadata into Spark:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
tokens = [token.as_dict() for token in w.token_management.list()]
display(spark.createDataFrame(tokens).orderBy("creation_time"))

Build a review queue with four columns:

token owner
token comment
current scopes
last used day

Then add the missing column yourself: workload owner. If the comment says test, old airflow, or do not delete, revoke or replace it. Those comments are archaeology, not operations.

For each live automation token, map the workload to API families. A deployment token for Databricks Asset Bundles might need jobs, workspace, pipelines, or unity-catalog depending on what it deploys. A BI extract token may only need sql. A service principal that creates other tokens needs authentication, and Databricks warns that tokens with the authentication scope can create new tokens with any scope.

That last one deserves a policy. Do not give authentication to a token just because a script failed once. Create a separate break glass path, log its use, and rotate it quickly.

If the inferred scopes are wrong, manually set scopes and opt out for that token. The PAT documentation says manually setting a token's scopes in the workspace UI or API permanently disables auto-scoping for that token.

curl -X PATCH https://<databricks-instance>/api/2.0/token/<token_id> \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "token": {
      "scopes": ["sql", "jobs", "unity-catalog"]
    },
    "update_mask": "scopes"
  }'

Databricks says scope changes can take up to 10 minutes to propagate. In practice, that means you should update scopes before a deployment window, not during a failing run.

What does Databricks PAT auto-scoping cost?

The clean answer: Databricks does not document a separate DBU meter for PAT auto-scoping in the release note or PAT docs. It is a security control on API access, not a warehouse, cluster, Lakeflow pipeline, Mosaic AI endpoint, or Databricks SQL query.

The messy answer: the bill can still move.

A failed job can retry on all purpose compute. A broken ingestion workflow can leave upstream data stale, causing expensive backfills. A disabled deployment token can pin engineers in manual recovery while a production warehouse keeps serving partial data. Those are not feature charges. They are consequence charges.

The admin controls around PATs do have plan implications. Databricks says managing personal access tokens in a workspace requires the Premium plan or above, and personal access token permissions are available only in the Premium plan or above.

There is also a token sprawl number worth using in your cleanup. Databricks says each PAT is valid for only one workspace, and a user can create up to 600 PATs per workspace. If that limit sounds absurdly high, remember what happens in a large platform: every migration, integration test, notebook export, and proof of concept leaves behind one more credential unless someone owns deletion.

Set a shorter maximum lifetime while you are here. Databricks documents maxTokenLifetimeDays, with a default maximum of 730 days, and shows this CLI setting for a 90 day maximum:

databricks workspace-conf set-status --json '{
  "maxTokenLifetimeDays": "90"
}'

That does two useful things. It reduces the blast radius of leaked tokens, and it forces teams to find the automation they forgot to migrate.

When should you keep PAT auto-scoping on, and when should you opt out?

Keep auto-scoping on for human generated long-lived tokens, BI tools with stable API needs, and older all APIs tokens you are actively cleaning up. It gives you a default path away from broad access without asking every analyst to understand the full API scope catalog.

Opt out, by manually setting scopes, when a token backs production automation with rare but legitimate operations. The key word is legitimate. If a token needs jobs once a quarter, document that need and set the scope. If nobody can explain why it needs authentication, remove it.

Use this decision rule:

If the workload is exploratory, leave auto-scoping on and keep the token short lived.
If the workload is production and predictable, set explicit scopes and monitor failures.
If the workload creates tokens, replace the pattern with OAuth or a tightly controlled service principal flow.
If the workload crosses account level administration, stop using PATs because Databricks says PATs cannot automate account level functionality.

The bigger migration is OAuth. Databricks says PATs are legacy authentication for many user account scenarios, and it recommends OAuth access tokens for stronger security and convenience. Auto-scoping makes PATs safer, but it should also make your remaining PAT inventory more visible.

That is the part worth doing this month. Do not wait for the 7 day enforcement email to discover that a token named airflow-prod-old2 deploys your most important Databricks SQL dashboard.

What should change in your token culture?

Databricks PAT auto-scoping turns least privilege from a policy slide into a platform default. Good.

Now make the human process match it. Every production token needs an owner, a workload name, a lifetime, a scope list, and a migration plan. If a token cannot survive that sentence, it should not survive the audit.

Sources

Databricks Lakeflow Designer GA cost guide for teams

2026-06-22T00:00:00Z

Databricks Lakeflow Designer has crossed the line from interesting demo to production tool, and that changes who gets to touch the transformation layer.

The feature is a visual, no-code canvas for preparing and transforming data inside Databricks. On June 16, 2026, Databricks said Lakeflow Designer is generally available, with workflows backed by production-ready code governed by Unity Catalog. The useful fact for a data engineer is simple: analysts can now build, preview, schedule, and hand off governed transformations without starting in a notebook.

That sounds like a power shift. It is also a bill shift. Lakeflow Designer uses Databricks compute, writes outputs to Unity Catalog, and can be scheduled through Lakeflow Jobs. If you influence the bill, your job is to make the canvas safe enough for analysts and observable enough for finance.

This guide is the engineering read: how it works, what it consumes, where governance actually applies, and where you should keep writing code. For the broader platform context, read our guide to how the Data Intelligence Platform splits work across governance, engineering, and AI.

What actually became GA in Lakeflow Designer?

Lakeflow Designer became generally available on June 16, 2026, after previously appearing in Databricks documentation as a preview feature. The release note says analysts can build visual transformation workflows using built-in operators or natural language, and that those workflows are backed by code governed by Unity Catalog.

The core object is a visual data prep file. Databricks describes each file as a directed acyclic graph made from operators such as filter, join, and transform, and the docs say those transformations are backed by code that can be versioned in Git and scheduled as jobs through Lakeflow Designer concepts. That matters because Designer is not a separate island for analyst experiments. It is a front end over code that can enter your normal production path.

The minimum requirements are tight enough to keep this from becoming spreadsheet sprawl. A workspace must have Unity Catalog enabled, and the user needs CAN USE permission on at least one general purpose compute resource, either serverless or all-purpose, according to the visual data prep setup requirements. That one permission check is where platform teams should start.

The GA scope also has one important availability caveat. The June release note says Lakeflow Designer is coming soon to workspaces with the compliance security profile enabled. If your regulated workspace depends on that profile, treat GA as a roadmap item until your account actually shows the feature.

The best mental model is this: Lakeflow Designer is a governed transformation workbench for analysts, not a replacement for Lakeflow Spark Declarative Pipelines. Use it to turn repeatable analyst prep into scheduled jobs. Keep complex streaming, CDC, strict test harnesses, and hand-tuned performance paths in engineering-owned code.

How do analysts build a workflow without creating a mess?

Every workflow starts with a Source operator. Databricks says the Source operator can browse Unity Catalog tables and volumes, browse workspace files, upload local CSV or Excel files, create a Unity Catalog table from a file, import from Google Drive, import from SharePoint, or use data previously landed through Lakeflow Connect as a table, according to the Designer ingestion documentation. That is a broad door, so permissions and landing zones matter.

The normal flow is short:

Add a Source operator.
Chain built-in operators such as Filter, Join, Aggregate, Pivot, SQL, or Transform.
Preview each operator output in the bottom pane.
Add an Output operator.
Write the result to a Unity Catalog table.
Schedule the visual data prep directly or add it as a task in a Lakeflow Job.

The preview behavior is where cost discipline begins. Databricks says operators process a limited sample by default, and the Rows scanned control can be set to Limit or Max; Max reruns upstream operators over the complete unbounded dataset and can take a long time, according to the build transformation guide. In plain English: the expensive button is visible to the analyst.

Designer also supports parameters. A SQL operator can reference a parameter with named marker syntax, while a Python operator can use dbutils.widgets.get(). This is the difference between one governed visual data prep and four copied canvases named final, final2, production, and real production.

SELECT
  customer_id,
  SUM(order_total) AS total_spend
FROM filtered_orders
WHERE environment = :environment
GROUP BY customer_id

That snippet is the kind of SQL you want analysts writing inside a SQL operator. It references the output of another operator by name, filtered_orders, and it keeps environment switching in a schedule parameter instead of hard-coding prod into the canvas.

The feature surface is bigger than a drag-and-drop filter builder. The built-in operator docs list 9 AI Function operators, 9 transformation operators, and 3 user-defined operator types. The chart below shows the useful boundary: analysts get enough built-ins to do real work, while engineers still own the extension points.

Source: Databricks Lakeflow Designer documentation. The docs list 9 AI Function operators, 9 transformation operators, and 3 user-defined operator types. Data Today benchmark.

Surface	What analysts get	Engineering control point
Built-in operators	Filter, Join, Aggregate, Pivot, Sort, SQL, Transform, Combine, Limit	Review naming, output tables, and schedule frequency
AI Function operator	`ai_gen`, `ai_extract`, `ai_classify`, `ai_translate`, and 5 more functions	Approve use cases, regions, and downstream review paths
User-defined operators	`python-run-function`, `uc-udf`, and `uc-udtf`	Own YAML, Unity Catalog functions, dependencies, and permissions

That table is the handoff contract. Analysts compose. Engineers set the safe vocabulary.

Where does Unity Catalog governance actually show up?

Unity Catalog is the reason Lakeflow Designer is more interesting than a BI tool's prep tab. The output operator writes results to a table in Unity Catalog, and its configuration requires a table name plus an output catalog and schema, according to the built-in operators documentation. That puts outputs inside the same privilege model, lineage surface, and catalog naming scheme you already use.

The governance split is cleanest for user-defined operators. Databricks supports exactly three user-defined operator types: python-run-function, uc-udf, and uc-udtf, and the docs state that Unity Catalog UDF and UDTF operators are governed by Unity Catalog permissions such as EXECUTE and USE SCHEMA in the user-defined operators guide. Use that to keep business logic reusable without turning every analyst into a package maintainer.

A simple Unity Catalog function can become a column-level operator:

CREATE OR REPLACE FUNCTION ops_shared.clean_zip(zip STRING)
RETURNS STRING
LANGUAGE SQL
RETURN regexp_replace(zip, '[^0-9]', '');

Then the platform team registers it as a uc-udf operator instead of asking every analyst to rediscover the same regex. That is the win: standard logic becomes a button, and the button still respects EXECUTE.

The YAML schema is also explicit. Databricks says every user-defined operator YAML file uses user-defined-operator-v0.1.0, and the root properties include fields such as schema, type, name, id, version, description, config, ports, run_function, and environment in the operator YAML reference. That gives you a reviewable artifact for Git, code owners, and change control.

schema: user-defined-operator-v0.1.0
type: uc-udf
name: Clean ZIP
id: ops.clean_zip
version: '1.0.0'
description: Standardize US ZIP codes before joins.

Keep that registration file in a Git folder. Designer files themselves appear in Git as notebooks with the file_name.designer.ipynb format, according to the visual data prep guide. That is awkward compared with a pure .py pipeline, but it is reviewable enough for analyst-owned transforms if you make code review part of promotion.

What does Lakeflow Designer cost or consume?

Lakeflow Designer does not remove compute from the equation. It moves compute behind a friendlier canvas.

For interactive work, the docs require CAN USE on a general purpose compute resource, either serverless or all-purpose. For production runs, Designer can create a scheduled job directly or be added as a task in a Lakeflow Job. Databricks says each Lakeflow Jobs task has an associated compute resource, and if you use serverless, Databricks configures that compute for you in the Lakeflow Jobs task documentation.

The practical billing rule is: preview spend lands where the interactive compute lands, and scheduled spend lands where the job compute lands. Databricks exposes serverless usage through system.billing.usage, including identity_metadata.run_as, usage_metadata.job_id, usage_metadata.job_name, notebook_id, and notebook_path, according to the serverless cost monitoring guide. Use that table before the first analyst enablement meeting, not after the invoice surprise.

SELECT
  identity_metadata.run_as,
  usage_metadata.job_id,
  usage_metadata.job_name,
  SUM(usage_quantity) AS total_dbu
FROM system.billing.usage
WHERE usage_unit = 'DBU'
  AND usage_date >= DATEADD(day, -30, current_date)
  AND billing_origin_product IN ('JOBS', 'INTERACTIVE')
GROUP BY 1, 2, 3
ORDER BY total_dbu DESC;

That query is intentionally broad. It will not magically label every visual canvas as Lakeflow Designer, but it will show which users and jobs are burning DBUs through the compute paths Designer uses. Add workspace tags, job naming conventions, and schedule names if you want showback that a finance partner can understand.

Also watch serverless quotas. Databricks says serverless quotas are measured in DBUs per hour, and serverless compute for notebooks, jobs, and Lakeflow Spark Declarative Pipelines has a per-workload scale-up limit in the serverless quota documentation. The same doc says quotas are a safety measure, not a general purpose spend limiter. Translation for platform owners: use budgets and alerts for dollars, not quotas.

AI Function operators need an extra review step. The Designer operator list includes functions such as ai_extract, ai_classify, ai_gen, and ai_translate. Those can be valuable for support-ticket tagging or text cleanup, but they can also turn a cheap row transform into a model workload. Approve them by use case, especially when the input table has millions of rows.

When should you use Designer instead of a notebook or pipeline?

Use Lakeflow Designer when the transformation is analyst-shaped: a join, filter, aggregate, reshape, or light SQL step that needs governance and scheduling. Do not force a data engineer to write a Spark job just to deduplicate a sales extract and publish a small gold table every morning.

A good first rollout has 3 rules:

Give analysts CAN USE on a constrained serverless or all-purpose compute target, not every cluster in the workspace.
Require Output operators to write into approved Unity Catalog schemas such as sandbox, analytics_dev, and analytics_prod.
Promote shared business logic through uc-udf and uc-udtf operators instead of copy-pasted SQL fragments.

Keep notebooks or Lakeflow Spark Declarative Pipelines for workloads with streaming state, CDC semantics, complex tests, dependency packaging, or data volumes where physical planning matters. Designer's Rows scanned: Max control is convenient, but it is not a performance model.

The wrong use case is also clear: do not use Designer as a shadow ETL tool for critical data contracts with no review path. A visual DAG can still break downstream users. The fact that it is pretty does not make it harmless.

The best operating model is a split ownership model. Analysts own the canvas logic and business intent. Data engineers own extension operators, Unity Catalog promotion paths, job schedules, and billing dashboards. That makes the tool productive without turning the warehouse into a craft fair.

Who should own the visual canvas now?

Lakeflow Designer GA is Databricks admitting that the transformation layer is no longer only an engineering surface. That is mostly good news for teams that are drowning in small requests.

The trade is control for throughput. You should take the trade where Unity Catalog can contain the blast radius, jobs can expose the spend, and Git can preserve review. You should reject the trade where visual convenience hides model calls, full-table previews, or production schemas with no owner.

The canvas belongs to analysts. The guardrails belong to you.

Sources

Databricks Data Intelligence Platform, explained for builders

2026-06-07T00:00:00Z

If you last touched Databricks as "the Spark notebooks company", the platform has grown into something much wider, and the bill grows with it. The Databricks Data Intelligence Platform is the lakehouse stack that sits on top of your own cloud storage and tries to be the one place a data team ingests, governs, queries, and runs AI on its data. This guide is the starting point for our Databricks studio: what the platform actually is, how a workload executes, and where the money goes.

The pitch is that a single platform on open Delta tables can replace the old split between a data lake for raw files and a warehouse for SQL. The interesting question for a builder is not the marketing, it is the plumbing: what runs where, what a DBU really is, and where the platform earns its keep versus where it quietly drains the budget.

What is the Data Intelligence Platform, concretely?

The platform is a managed control plane that Databricks hosts, connected to compute and storage that run in your own cloud account on AWS, Azure, or GCP. Your data sits in your cloud object storage as Delta Lake tables, an open format built on Parquet with a transaction log that adds ACID transactions, time travel, and schema enforcement to plain files.

Four layers stack on top of that storage, and it helps to hold them separate in your head:

Storage: Delta tables in your object store, the open foundation everything else reads and writes.
Governance: Unity Catalog, the single metastore that owns permissions, lineage, and discovery across every workspace.
Compute: clusters, SQL warehouses, and serverless, the engines that actually do work and burn money.
Intelligence: Mosaic AI, the model serving, AI functions, and agent tooling layered across the rest.

The architectural fact worth internalizing is that storage and compute are fully decoupled. Your tables persist whether or not anything is running, and you pay for compute only while an engine is on. That single design choice explains most of the platform's cost behaviour, which we will come back to.

How does a workload actually run?

Every piece of work runs on compute you choose, and picking the wrong type is the most common and most expensive beginner mistake. There are three broad families.

Compute type	What it is for	Billing shape
All-purpose clusters	Interactive notebooks and ad hoc exploration	Highest DBU rate, easy to leave running
Job clusters	Scheduled pipelines and production jobs	Lower DBU rate, spun up and torn down per run
SQL warehouses	BI and SQL queries on Databricks SQL	Sized in T-shirt sizes, autoscale on concurrency

The unit on the Databricks side of the bill is the DBU, a Databricks Unit, which meters processing per second at a rate that depends on the compute type and tier. Crucially, you pay that DBU charge on top of the underlying cloud VM cost, which your cloud provider bills you separately. So a cluster left running overnight costs you twice: once in idle DBUs and once in idle EC2 or equivalent.

The single most common cost surprise is an all-purpose cluster used for a job that should have run on a cheaper job cluster, often with auto-termination disabled. Moving that workload to a job cluster with a short auto-termination window is usually the highest-return change a new team can make.

Where does the cost actually go?

This is the question that decides whether a Databricks workspace stays affordable, and the answer has layers you pay separately. The chart below shows the rough shape of a typical monthly spend: compute DBUs dominate at around 65 percent, the underlying cloud VMs are roughly 25 percent, and storage and egress are about 10 percent. The exact mix moves with your workload, but the lesson holds: compute, not storage, is where the bill lives.

Illustrative: a typical split of monthly Databricks spend across compute DBUs, underlying cloud VMs, and storage. Data Today.

The good news is that the platform now bills its own usage into system tables you can query directly. The system.billing.usage view in Unity Catalog records DBU consumption per workload, so you can attribute spend instead of guessing.

SELECT
  usage_metadata.job_id,
  sku_name,
  SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY usage_metadata.job_id, sku_name
ORDER BY dbus DESC
LIMIT 20;

That one query tells you which jobs are actually expensive, which is where any FinOps effort should start. We treat cost as its own recurring topic for exactly this reason.

How does Unity Catalog change governance?

Unity Catalog is the platform's governance layer, and it is the piece that turns a pile of workspaces into one governed estate. It uses a three-level namespace, catalog.schema.table, so a fully qualified name like prod.sales.orders means the same thing in every workspace attached to the metastore.

Grants are standard SQL, which makes access reviews legible:

GRANT SELECT ON TABLE prod.sales.orders TO `analysts`;

Beyond plain grants, Unity Catalog tracks column and table lineage automatically, enforces row filters and column masks, and powers Delta Sharing for handing governed data to another organization without copying it. If you run more than one workspace, getting the catalog design right early is the decision that ages best, well before you tune a single query.

Where does Mosaic AI fit?

Mosaic AI is Databricks' name for the AI features layered across the platform: Model Serving for hosting models behind an endpoint, Vector Search for retrieval, agent tooling like Agent Bricks, and SQL-native AI functions. The last of these is the easiest on-ramp, because you can call a model straight from a query:

SELECT ai_query(
  'databricks-meta-llama-3-3-70b-instruct',
  'Summarize this ticket: ' || ticket_body
) AS summary
FROM support.tickets
LIMIT 100;

The honest read is that this is the fastest-moving and least settled part of the platform, which is precisely why it deserves sceptical coverage rather than hype. The practical stance for a builder is to let these functions accelerate well-bounded tasks, summarization, extraction, classification, while keeping a human reviewing anything that touches a customer or a production write. We will track each Mosaic AI capability as it ships and judge whether it is genuinely production-ready or still a demo.

What should you do with this?

If you are evaluating or running the Data Intelligence Platform, a few principles travel well:

Match compute to the workload. Job clusters for jobs, SQL warehouses for BI, serverless where the startup latency hurts. Never run production on an all-purpose cluster by default.
Set auto-termination everywhere. Idle compute is the most common line of waste, and it bills twice.
Query system.billing.usage. Attribute DBUs to jobs before you try to optimize anything.
Design Unity Catalog early. The three-level namespace and lineage pay off most when set before the estate sprawls.
Adopt Mosaic AI deliberately. Use it where review is cheap; gate it where mistakes are expensive.

This guide is the foundation. From here the studio goes deeper on each piece: Lakeflow pipelines and ingestion, Databricks SQL warehouses and materialized views, Unity Catalog governance patterns, the compute and FinOps habits that keep DBUs in check, and the Mosaic AI features as they land. The platform is moving quickly, and the goal here is the same as everywhere on Data Today: tell you what actually changed and what it means for the thing you are building.