Azure Databricks Cost Optimization: Autoscaling, Spot Instances, Photon, and Cluster Policies That Actually Save Money
Databricks bills by DBU (Databricks Unit). DBUs add up fast when you have multiple teams running notebooks, jobs, and SQL warehouses all day. The good news is that most Databricks cost problems are architectural, not contractual — the wrong cluster type for the workload, no autoscaling, spot instances turned off, or warehouses left running overnight.
In this post I'll walk through the most impactful cost levers on Databricks with concrete configuration examples for each.
Where Databricks Cost Comes From
Cost Optimization Decision Flow
Cost Lever Comparison
| Lever | Typical Saving | Best For | Risk |
|---|---|---|---|
| Spot / Preemptible VMs | 60-80% VM cost | Job clusters, batch ETL | Spot interruption (mitigated by checkpoints) |
| Autoscaling | 30-60% idle waste | Interactive clusters, variable workloads | Slow scale-up on sudden spikes |
| Photon Engine | 2-4x faster runtime = fewer hours | SQL, Delta reads, ETL | Extra DBU cost per hour (net positive on CPU-heavy workloads) |
| SQL Warehouse auto-stop | 100% idle cost | BI / SQL analytics | Cold start latency on first query |
| Cluster policies | Prevents over-provisioning | All cluster types | Requires admin setup |
Trigger.AvailableNow() | Eliminates always-on streaming | Near-real-time batch | Higher latency than continuous streaming |
| Instance pools | Faster startup + cheaper VMs | Frequent short jobs | Pool idle VMs still cost money |
Step 1 — Cluster Policies: Stop Engineers Spinning Up 32-Node Clusters
Cluster policies are JSON rules that constrain what engineers can configure. Without them, someone will always spin up a Standard_D32s_v3 cluster for a notebook that reads 1000 rows.
{
"name": "Data Engineer Standard",
"definition": {
"spark_version": {
"type": "allowlist",
"values": ["14.3.x-scala2.12", "15.1.x-scala2.12"],
"defaultValue": "14.3.x-scala2.12"
},
"node_type_id": {
"type": "allowlist",
"values": ["Standard_DS3_v2", "Standard_DS4_v2", "Standard_DS5_v2"],
"defaultValue": "Standard_DS3_v2"
},
"autoscale.min_workers": {
"type": "range",
"minValue": 1,
"maxValue": 4,
"defaultValue": 2
},
"autoscale.max_workers": {
"type": "range",
"minValue": 2,
"maxValue": 8,
"defaultValue": 4
},
"aws_attributes.availability": {
"type": "fixed",
"value": "SPOT_WITH_FALLBACK"
},
"autotermination_minutes": {
"type": "range",
"minValue": 10,
"maxValue": 60,
"defaultValue": 30
},
"custom_tags.team": {
"type": "unlimited",
"isOptional": false
},
"custom_tags.cost_center": {
"type": "unlimited",
"isOptional": false
}
}
}The team and cost_center tags are required — without them you can't attribute spend in your cloud bill.
Step 2 — Autoscaling Job Clusters with Spot Fallback
Job clusters should always use SPOT_WITH_FALLBACK. Spot for cost, on-demand fallback so the job doesn't fail if spot capacity is unavailable.
# Databricks Jobs API — optimized job cluster config
job_cluster_config = {
"job_cluster_key": "etl-optimized",
"new_cluster": {
"spark_version": "14.3.x-scala2.12",
"node_type_id": "Standard_DS4_v2",
"driver_node_type_id": "Standard_DS3_v2", # smaller driver saves cost
"autoscale": {
"min_workers": 2,
"max_workers": 16,
},
"azure_attributes": {
"availability": "SPOT_WITH_FALLBACK_AZURE",
"spot_bid_max_price": -1, # -1 = on-demand price cap (safe default)
"first_on_demand": 1, # driver is always on-demand
},
"spark_conf": {
"spark.sql.shuffle.partitions": "auto",
"spark.databricks.io.cache.enabled": "true", # disk cache
"spark.databricks.delta.optimizeWrite.enabled": "true",
},
"runtime_engine": "PHOTON", # enable Photon for ETL workloads
"custom_tags": {
"team": "data-engineering",
"cost_center": "cc-1042",
"workload": "batch-etl",
},
}
}Step 3 — SQL Warehouse Configuration for BI Workloads
SQL Warehouses are the biggest hidden cost source for BI teams. They auto-start on query but won't auto-stop unless you configure it. A warehouse left on overnight burns DBUs for 8 hours of idle time.
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import CreateWarehouseRequestWarehouseType
w = WorkspaceClient()
warehouse = w.warehouses.create(
name="bi-team-warehouse",
cluster_size="Small", # start small, scale up only if needed
min_num_clusters=1,
max_num_clusters=3, # scale out on concurrent queries
auto_stop_mins=15, # stop after 15 min idle — critical
warehouse_type=CreateWarehouseRequestWarehouseType.PRO,
enable_photon=True,
channel={"name": "CHANNEL_NAME_CURRENT"},
tags={
"custom_tags": [
{"key": "team", "value": "bi"},
{"key": "cost_center", "value": "cc-2011"},
]
}
)
print(f"Warehouse created: {warehouse.id}")Step 4 — Cost Monitoring with Databricks System Tables
Unity Catalog's system tables include billing data you can query directly. Build this into a daily Slack alert or Databricks SQL dashboard.
# Top 10 most expensive workloads in the last 30 days
cost_df = spark.sql("""
SELECT
usage_metadata.job_id AS job_id,
usage_metadata.notebook_id AS notebook_id,
identity_metadata.run_as AS run_as_user,
custom_tags['team'] AS team,
custom_tags['cost_center'] AS cost_center,
SUM(usage_quantity) AS total_dbus,
ROUND(SUM(usage_quantity) * 0.40, 2) AS estimated_cost_usd,
COUNT(DISTINCT usage_date) AS active_days
FROM system.billing.usage
WHERE
usage_date >= current_date() - INTERVAL 30 DAYS
AND sku_name LIKE '%ALL_PURPOSE%'
GROUP BY 1, 2, 3, 4, 5
ORDER BY total_dbus DESC
LIMIT 10
""")
cost_df.show(truncate=False)
# Clusters without cost tags — these are unattributed spend
untagged_df = spark.sql("""
SELECT
usage_date,
cluster_id,
SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE
usage_date >= current_date() - INTERVAL 7 DAYS
AND (custom_tags['team'] IS NULL OR custom_tags['cost_center'] IS NULL)
GROUP BY 1, 2
ORDER BY dbus DESC
""")
print(f"Untagged DBU consumption in last 7 days:")
untagged_df.show()Things to Watch in Production
Photon costs more per DBU but fewer DBUs overall. Photon adds a DBU premium but runs workloads 2-4x faster on SQL and ETL. Do the math on your specific workload — for CPU-heavy jobs it almost always comes out cheaper total.
first_on_demand: 1 keeps your driver on-demand. If the driver node is spot and gets interrupted mid-job, you lose everything. Keep the driver on on-demand by setting first_on_demand: 1 in your cluster config.
Delta cache is free on DS-series VMs. The Databricks disk cache (spark.databricks.io.cache.enabled) uses the local SSD on DS-series Azure VMs to cache Delta files. It's free and speeds up repeated reads significantly — always enable it.
Tag enforcement pays off in month 2. The first month of tagging feels like overhead. By month 2 you'll have a clear breakdown of which team is spending what, which makes budget conversations much easier and catches runaway workloads before they hit the invoice.
Wrapping Up
Databricks cost optimization is mostly about defaults. The default cluster has no autoscaling, no spot instances, no auto-termination, and no cost tags. Change all four and you'll cut your bill materially before you even look at workload-level tuning. Layer in Photon for SQL-heavy workloads, SQL Warehouse auto-stop for BI teams, and system table monitoring to catch what slips through, and you have a cost posture that scales with your platform.