Jubin Soni - Portfolio & Blog

The default Azure Databricks workspace is more open than most security teams are comfortable with. Control plane traffic goes over the public internet, secrets get hardcoded in notebooks, and there's no enforced MFA on workspace access. None of that is acceptable in a production enterprise environment.

In this post I'll walk through locking down Azure Databricks properly: VNet injection with Private Link to eliminate public internet exposure, Azure Active Directory integration for identity, and Azure Key Vault for secret management — all configured in a way that actually holds up to a security audit.

Security Architecture Overview

Auth and Network Request Flow

Security Controls Comparison

Control	Default State	Hardened State	Risk Mitigated
Network access	Public internet	Private Link + VNet injection	Data exfiltration, MITM
Authentication	Password only	AAD + MFA + Conditional Access	Credential compromise
Secret storage	Hardcoded in notebooks	Azure Key Vault secret scope	Secret leakage in Git
Data plane traffic	Public ADLS endpoint	Private Endpoint	Network interception
Cluster internet access	Unrestricted	No public IP + egress firewall	Data exfiltration
Admin access	Any workspace admin	AAD group + PIM just-in-time	Privilege escalation
Audit logging	Minimal	Diagnostic settings + Sentinel	Incident detection

Step 1 — VNet Injection: Keep the Data Plane in Your Network

VNet injection puts the Databricks data plane (cluster VMs) inside your own VNet. Without it, your clusters run in a Databricks-managed VNet you have no control over.

bash

# Create the VNet and subnets required for Databricks VNet injection
# Databricks requires two dedicated subnets: public and private

RESOURCE_GROUP="rg-databricks-prod"
VNET_NAME="vnet-databricks-prod"
LOCATION="eastus"

az network vnet create \
  --resource-group $RESOURCE_GROUP \
  --name $VNET_NAME \
  --location $LOCATION \
  --address-prefix "10.10.0.0/16"

# Public subnet (cluster nodes outbound, no public IPs)
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name $VNET_NAME \
  --name "databricks-public" \
  --address-prefix "10.10.1.0/24" \
  --delegations "Microsoft.Databricks/workspaces" \
  --network-security-group "nsg-databricks-public"

# Private subnet (cluster nodes inbound)
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name $VNET_NAME \
  --name "databricks-private" \
  --address-prefix "10.10.2.0/24" \
  --delegations "Microsoft.Databricks/workspaces" \
  --network-security-group "nsg-databricks-private"

# Deploy Databricks workspace with VNet injection
az databricks workspace create \
  --resource-group $RESOURCE_GROUP \
  --name "adb-prod" \
  --location $LOCATION \
  --sku premium \
  --custom-virtual-network-id $(az network vnet show -g $RESOURCE_GROUP -n $VNET_NAME --query id -o tsv) \
  --custom-public-subnet-name "databricks-public" \
  --custom-private-subnet-name "databricks-private" \
  --no-public-ip true   # disable public IPs on cluster nodes

Step 2 — Private Link for Control Plane Traffic

Private Link routes Databricks control plane traffic (API calls, notebook connections) through a private endpoint in your VNet instead of the public internet.

bash

# Create Private Endpoint for Databricks control plane
WORKSPACE_ID=$(az databricks workspace show \
  --resource-group $RESOURCE_GROUP \
  --name "adb-prod" \
  --query id -o tsv)

az network private-endpoint create \
  --resource-group $RESOURCE_GROUP \
  --name "pe-databricks-auth" \
  --vnet-name $VNET_NAME \
  --subnet "databricks-private" \
  --private-connection-resource-id $WORKSPACE_ID \
  --group-id "databricks_ui_api" \
  --connection-name "conn-databricks-auth" \
  --location $LOCATION

# Create Private DNS Zone so internal DNS resolves to private endpoint
az network private-dns zone create \
  --resource-group $RESOURCE_GROUP \
  --name "privatelink.azuredatabricks.net"

az network private-dns link vnet create \
  --resource-group $RESOURCE_GROUP \
  --zone-name "privatelink.azuredatabricks.net" \
  --name "dns-link-databricks" \
  --virtual-network $VNET_NAME \
  --registration-enabled false

az network private-endpoint dns-zone-group create \
  --resource-group $RESOURCE_GROUP \
  --endpoint-name "pe-databricks-auth" \
  --name "databricks-dns-zone-group" \
  --private-dns-zone "privatelink.azuredatabricks.net" \
  --zone-name "databricks"

Step 3 — Azure Key Vault Secret Scope in Databricks

Databricks secret scopes backed by Azure Key Vault mean secrets never live in notebooks, environment variables, or Databricks-managed storage.

python

# In a notebook: access secrets via Key Vault-backed secret scope
# The scope is configured once at workspace level (via Databricks Secrets API)
# After that, every notebook just calls dbutils.secrets.get()

# Example: access storage credentials
storage_account = "yourstorage"
tenant_id     = dbutils.secrets.get(scope="kv-prod-scope", key="adb-sp-tenant-id")
client_id     = dbutils.secrets.get(scope="kv-prod-scope", key="adb-sp-client-id")
client_secret = dbutils.secrets.get(scope="kv-prod-scope", key="adb-sp-client-secret")

# Configure ADLS access using the fetched credentials
spark.conf.set(
    f"fs.azure.account.oauth2.client.id.{storage_account}.dfs.core.windows.net",
    client_id
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.secret.{storage_account}.dfs.core.windows.net",
    client_secret
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.endpoint.{storage_account}.dfs.core.windows.net",
    f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
)

# Verify the secret scope is working (never print the actual value)
print(f"Secret scope active. Client ID length: {len(client_id)}")
print(f"Storage account: {storage_account}")

bash

# Create the Key Vault-backed secret scope via Databricks CLI
# Run this once during workspace setup

databricks secrets create-scope \
  --scope kv-prod-scope \
  --scope-backend-type AZURE_KEYVAULT \
  --resource-id $(az keyvault show --name "kv-databricks-prod" --query id -o tsv) \
  --dns-name "https://kv-databricks-prod.vault.azure.net/"

Step 4 — Audit Logging via Diagnostic Settings

Send all Databricks audit events to Log Analytics or Azure Sentinel for SIEM integration and compliance reporting.

bash

# Enable Diagnostic Settings on the Databricks workspace
WORKSPACE_ID=$(az databricks workspace show \
  --resource-group $RESOURCE_GROUP \
  --name "adb-prod" --query id -o tsv)

LOG_ANALYTICS_ID=$(az monitor log-analytics workspace show \
  --resource-group $RESOURCE_GROUP \
  --workspace-name "law-security-prod" --query id -o tsv)

az monitor diagnostic-settings create \
  --resource $WORKSPACE_ID \
  --name "adb-audit-logs" \
  --workspace $LOG_ANALYTICS_ID \
  --logs '[
    {"category": "dbfs", "enabled": true},
    {"category": "clusters", "enabled": true},
    {"category": "accounts", "enabled": true},
    {"category": "jobs", "enabled": true},
    {"category": "notebook", "enabled": true},
    {"category": "secrets", "enabled": true},
    {"category": "sqlPermissions", "enabled": true},
    {"category": "workspace", "enabled": true}
  ]'

python

# Query audit logs in Log Analytics to detect secret access anomalies
# Run in Log Analytics workspace
query = """
DatabricksSecrets
| where TimeGenerated > ago(24h)
| where ActionName == "getSecret"
| summarize AccessCount = count() by UserName, SecretScope, bin(TimeGenerated, 1h)
| where AccessCount > 50   // flag unusual secret access volume
| order by AccessCount desc
"""
# Paste this KQL into Log Analytics or Azure Sentinel
print("KQL query ready for Log Analytics:")
print(query)

Things to Watch in Production

No-public-IP clusters still need egress for Maven / PyPI. With --no-public-ip true, cluster nodes can't reach the internet. If your init scripts install packages from PyPI or Maven Central, you need an Azure Firewall or NAT Gateway allowing outbound HTTPS to those domains specifically. Alternatively, mirror packages in Azure Artifacts or a private Nexus.

Private Link doesn't protect the Databricks web app by default. Private Link secures API and cluster traffic but the workspace UI is still accessible publicly unless you configure publicNetworkAccess: Disabled on the workspace and route browser traffic through VPN or ExpressRoute.

Key Vault network rules must allow the Databricks subnet. If your Key Vault has Allow access from: Selected networks enabled, add the Databricks private subnet to the allowed list, or use a Key Vault Private Endpoint in the same VNet.

Rotate service principal secrets on a schedule. Key Vault supports secret rotation with Event Grid triggers. Set up a rotation function that generates a new SP client secret, writes it to Key Vault, and updates the AAD app registration — all without touching any Databricks notebook.

Wrapping Up

Security on Azure Databricks is an onion: VNet injection and Private Link eliminate network-level exposure, AAD + Conditional Access handles identity, Key Vault removes secrets from code, and diagnostic settings give you the audit trail. None of these is optional in a production enterprise deployment handling sensitive data. The good news is all of them are well-supported by Azure and Databricks — it's a matter of configuration, not custom development.

Azure Databricks Security: Private Link, Azure AD, and Key Vault Done Right

Security Architecture Overview

Auth and Network Request Flow

Security Controls Comparison

Step 1 — VNet Injection: Keep the Data Plane in Your Network

Step 2 — Private Link for Control Plane Traffic

Step 3 — Azure Key Vault Secret Scope in Databricks

Step 4 — Audit Logging via Diagnostic Settings

Things to Watch in Production

Wrapping Up

References