← All posts

Azure Databricks + Azure Data Factory and ADLS Gen2: Building a Production Data Integration Pipeline

7 min read4.5k
#Azure#Databricks#DataEngineering#DeltaLake#Python#Workflows#Spark#Data

Azure Databricks doesn't live in isolation. In most enterprise Azure environments it sits downstream of Azure Data Factory, which orchestrates ingestion from databases, SaaS tools, and on-premises sources, and it reads and writes data from Azure Data Lake Storage Gen2. Getting these three services to work together cleanly — with the right authentication, the right data formats, and proper orchestration handoffs — is what separates a PoC from a production data platform.

In this post I'll walk through wiring up ADF, ADLS Gen2, and Azure Databricks end to end: ADF pipelines that trigger Databricks notebooks, Databricks reading and writing Delta Lake on ADLS Gen2 via service principal auth, and monitoring the whole thing in one place.

Architecture Overview

Integration Flow

Authentication Options Compared

MethodBest ForSetup ComplexitySecurity
Service Principal + Key VaultProduction pipelinesMediumHigh — no shared keys
Managed IdentityADF → ADLS directLowHigh — no credentials at all
Storage Account KeyDev / PoC onlyLowLow — rotates manually
User credential passthroughInteractive notebooksLowMedium — tied to user session
Unity Catalog External LocationUnity Catalog workspacesMediumHighest — governed by UC

Step 1 — Configure ADLS Gen2 Access in Databricks via Service Principal

Never use storage account keys in production. Use a service principal registered in Azure AD and store the secret in Azure Key Vault.

python
# Set ADLS Gen2 access via Service Principal (OAuth 2.0)
# Secrets come from Databricks secret scope backed by Azure Key Vault

tenant_id    = dbutils.secrets.get(scope="kv-scope", key="sp-tenant-id")
client_id    = dbutils.secrets.get(scope="kv-scope", key="sp-client-id")
client_secret= dbutils.secrets.get(scope="kv-scope", key="sp-client-secret")
storage_name = "yourstorage"

spark.conf.set(f"fs.azure.account.auth.type.{storage_name}.dfs.core.windows.net",
               "OAuth")
spark.conf.set(f"fs.azure.account.oauth.provider.type.{storage_name}.dfs.core.windows.net",
               "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set(f"fs.azure.account.oauth2.client.id.{storage_name}.dfs.core.windows.net",
               client_id)
spark.conf.set(f"fs.azure.account.oauth2.client.secret.{storage_name}.dfs.core.windows.net",
               client_secret)
spark.conf.set(f"fs.azure.account.oauth2.client.endpoint.{storage_name}.dfs.core.windows.net",
               f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")

# Test access
files = dbutils.fs.ls(f"abfss://raw@{storage_name}.dfs.core.windows.net/")
print(f"Connected. {len(files)} objects in raw container.")

Step 2 — Read ADF-Landed Files and Write Delta

ADF lands raw CSV/JSON/Parquet files in the raw zone. Databricks picks them up, transforms, and writes Delta back to the curated zone.

python
from pyspark.sql.functions import current_timestamp, lit, input_file_name
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, TimestampType

STORAGE    = "yourstorage"
RAW_PATH   = f"abfss://raw@{STORAGE}.dfs.core.windows.net/sales/"
DELTA_PATH = f"abfss://curated@{STORAGE}.dfs.core.windows.net/delta/sales/"

# Read Parquet files landed by ADF Copy Activity
raw_df = (
    spark.read
        .format("parquet")
        .option("mergeSchema", "true")
        .load(RAW_PATH)
        .withColumn("_ingested_at",  current_timestamp())
        .withColumn("_source_file",  input_file_name())
        .withColumn("_pipeline_run", lit(dbutils.widgets.get("adf_pipeline_run_id")))
)

print(f"Rows read from ADF landing zone: {raw_df.count()}")

# Clean and write Bronze Delta
bronze_df = raw_df \
    .dropDuplicates(["order_id"]) \
    .filter(raw_df.order_id.isNotNull())

bronze_df.write \
    .format("delta") \
    .mode("append") \
    .option("mergeSchema", "true") \
    .save(DELTA_PATH)

# Register in Unity Catalog
spark.sql(f"""
    CREATE TABLE IF NOT EXISTS production.sales.bronze_orders
    USING DELTA
    LOCATION '{DELTA_PATH}'
""")

# Pass row count back to ADF via notebook exit value
row_count = bronze_df.count()
dbutils.notebook.exit(str(row_count))

Step 3 — ADF Pipeline with Databricks Notebook Activity

In ADF, a Notebook Activity calls the Databricks notebook and waits for it to complete. Pass ADF pipeline context (run ID, trigger time) as notebook parameters for lineage tracking.

json
{
  "name": "IngestAndTransformSales",
  "properties": {
    "activities": [
      {
        "name": "CopyRawSalesFiles",
        "type": "Copy",
        "typeProperties": {
          "source": {
            "type": "AzureSqlSource",
            "sqlReaderQuery": "SELECT * FROM dbo.sales WHERE updated_at > '@{pipeline().parameters.watermark}'"
          },
          "sink": {
            "type": "ParquetSink",
            "storeSettings": {
              "type": "AzureBlobFSWriteSettings"
            }
          }
        },
        "inputs":  [{ "referenceName": "SqlServerLinkedService",  "type": "LinkedServiceReference" }],
        "outputs": [{ "referenceName": "ADLSRawSalesDataset",     "type": "DatasetReference" }]
      },
      {
        "name": "RunDatabricksNotebook",
        "type": "DatabricksNotebook",
        "dependsOn": [{ "activity": "CopyRawSalesFiles", "dependencyConditions": ["Succeeded"] }],
        "typeProperties": {
          "notebookPath": "/pipelines/sales_bronze_transform",
          "baseParameters": {
            "adf_pipeline_run_id": { "value": "@pipeline().RunId",       "type": "Expression" },
            "adf_trigger_time":    { "value": "@pipeline().TriggerTime", "type": "Expression" },
            "env":                 { "value": "@pipeline().parameters.env" }
          }
        },
        "linkedServiceName": { "referenceName": "AzureDatabricksLinkedService", "type": "LinkedServiceReference" }
      },
      {
        "name": "CheckRowCount",
        "type": "IfCondition",
        "dependsOn": [{ "activity": "RunDatabricksNotebook", "dependencyConditions": ["Succeeded"] }],
        "typeProperties": {
          "expression": {
            "value": "@greater(int(activity('RunDatabricksNotebook').output.runOutput), 0)",
            "type": "Expression"
          },
          "ifFalseActivities": [
            {
              "name": "AlertZeroRows",
              "type": "WebActivity",
              "typeProperties": {
                "url": "https://your-slack-webhook-url",
                "method": "POST",
                "body": { "text": "WARNING: Databricks notebook returned 0 rows for pipeline @{pipeline().RunId}" }
              }
            }
          ]
        }
      }
    ],
    "parameters": {
      "watermark": { "type": "string" },
      "env":       { "type": "string", "defaultValue": "production" }
    }
  }
}

Step 4 — Mount ADLS Gen2 as a Databricks Mount Point (Legacy) vs ABFS Direct Access

Older Databricks setups use mount points (/mnt/raw). Modern best practice is direct ABFS paths with Unity Catalog External Locations — no mounting needed.

python
# LEGACY: Mount point approach (still works, not recommended for new setups)
dbutils.fs.mount(
    source="abfss://raw@yourstorage.dfs.core.windows.net/",
    mount_point="/mnt/raw",
    extra_configs={
        "fs.azure.account.auth.type":                "OAuth",
        "fs.azure.account.oauth.provider.type":      "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
        "fs.azure.account.oauth2.client.id":         client_id,
        "fs.azure.account.oauth2.client.secret":     client_secret,
        "fs.azure.account.oauth2.client.endpoint":   f"https://login.microsoftonline.com/{tenant_id}/oauth2/token",
    }
)

# MODERN: Unity Catalog External Location (recommended)
# Set up once via SQL, then reference by catalog path
spark.sql("""
    CREATE EXTERNAL LOCATION raw_zone
    URL 'abfss://raw@yourstorage.dfs.core.windows.net/'
    WITH (STORAGE CREDENTIAL your_storage_credential)
    COMMENT 'Raw landing zone for ADF-ingested files';
""")

# Then just use it by path — no mount needed
df = spark.read.parquet("abfss://raw@yourstorage.dfs.core.windows.net/sales/")

Things to Watch in Production

ADF Notebook Activity timeout defaults to 12 hours. For long-running Databricks jobs, this is fine. But for notebooks expected to complete in 10 minutes, set an explicit timeout of 15-20 minutes so a hung notebook doesn't block your ADF pipeline for half a day.

Pass ADF run ID into every Databricks notebook. Store it as an MLflow tag or Delta column. When something goes wrong at 2am you want to trace a bad Delta write back to the exact ADF pipeline run that triggered it.

Avoid mount points for new Unity Catalog workspaces. Mounts bypass Unity Catalog governance — they're not subject to row filters or column masks. Use ABFS paths with External Locations for all new development.

Use ADF's retry policy on the Notebook Activity. Transient Databricks cluster startup failures happen. Set retryCount: 2 and retryIntervalInSeconds: 60 on the Notebook Activity so a slow cluster warm-up doesn't fail the whole pipeline.

Wrapping Up

ADF + ADLS Gen2 + Azure Databricks is one of the most common enterprise data platform patterns on Azure. ADF handles the connectivity and scheduling complexity for pulling from hundreds of source systems; Databricks handles the transformation and feature engineering; ADLS Gen2 is the durable, cost-effective store that ties them together. Get the authentication right (service principal + Key Vault), pass pipeline context between services for lineage, and use ABFS paths over mount points for any Unity Catalog workspace.

References