Databricks Unity Catalog: Data Governance and Access Control at Scale

The hardest part of scaling a data platform isn't the compute. It's knowing who can access what, proving it to auditors, and not breaking everything when someone leaves the team. Databricks Unity Catalog is the answer to that problem — a unified governance layer that sits across all your Databricks workspaces and gives you centralized access control, data lineage, and auditing in one place.

In this post I'll walk through setting up Unity Catalog on Databricks, configuring fine-grained access control, and using it to enforce row-level and column-level security on sensitive datasets.

What is Databricks Unity Catalog?

Unity Catalog is a metastore layer that unifies governance across all Databricks workspaces in an Azure tenant. Before Unity Catalog, each workspace had its own Hive metastore, its own permissions, and zero visibility into what was happening in other workspaces. Unity Catalog replaces all of that with a single control plane.

Key capabilities:

• Three-level namespace: catalog.schema.table
• Row-level and column-level security
• Automatic data lineage across notebooks, jobs, and SQL queries
• Centralized audit logs via system tables
• Fine-grained attribute-based access control (ABAC)

Architecture Overview

Access Control Flow

Unity Catalog Object Hierarchy

Step 1 — Setting Up the Three-Level Namespace

-- Create catalogs for each environment
CREATE CATALOG IF NOT EXISTS production
  COMMENT 'Production data — restricted access';

CREATE CATALOG IF NOT EXISTS development
  COMMENT 'Development sandbox — engineers only';

-- Create schemas within the catalog
USE CATALOG production;

CREATE SCHEMA IF NOT EXISTS orders
  COMMENT 'Order transaction data'
  MANAGED LOCATION 'abfss://unity@yourstorage.dfs.core.windows.net/production/orders/';

CREATE SCHEMA IF NOT EXISTS customers
  COMMENT 'Customer PII data — restricted'
  MANAGED LOCATION 'abfss://unity@yourstorage.dfs.core.windows.net/production/customers/';

-- Grant schema-level access to a group
GRANT USAGE ON CATALOG production TO `data-analysts`;
GRANT USAGE ON SCHEMA production.orders TO `data-analysts`;
GRANT SELECT ON TABLE production.orders.transactions TO `data-analysts`;

-- Analysts can read orders but NOT customers
REVOKE ALL PRIVILEGES ON SCHEMA production.customers FROM `data-analysts`;

Step 2 — Column Masking for PII

Column masks let you return a transformed value instead of the raw column based on who is querying. A data analyst sees ****-****-****-1234 while a data engineer sees the full card number.

-- Create a masking function
CREATE OR REPLACE FUNCTION production.customers.mask_card_number(card_number STRING)
  RETURNS STRING
  RETURN CASE
    WHEN is_account_group_member('pci-engineers') THEN card_number
    ELSE CONCAT('****-****-****-', RIGHT(card_number, 4))
  END;

-- Apply the mask to the column
ALTER TABLE production.customers.payments
  ALTER COLUMN credit_card_number
  SET MASK production.customers.mask_card_number;

-- Same for email — partial mask for non-PII teams
CREATE OR REPLACE FUNCTION production.customers.mask_email(email STRING)
  RETURNS STRING
  RETURN CASE
    WHEN is_account_group_member('pii-engineers') THEN email
    ELSE CONCAT(LEFT(email, 2), '***@***.com')
  END;

ALTER TABLE production.customers.profiles
  ALTER COLUMN email
  SET MASK production.customers.mask_email;

Step 3 — Row-Level Security by Region

Row filters restrict which rows a user can see based on their group membership. A European analyst should only see European customer data.

# First create the row filter function in SQL
spark.sql("""
  CREATE OR REPLACE FUNCTION production.customers.region_filter(region STRING)
    RETURNS BOOLEAN
    RETURN CASE
      WHEN is_account_group_member('global-admins') THEN TRUE
      WHEN is_account_group_member('eu-analysts')   THEN region = 'EU'
      WHEN is_account_group_member('us-analysts')   THEN region = 'US'
      WHEN is_account_group_member('apac-analysts') THEN region = 'APAC'
      ELSE FALSE
    END;
""")

# Apply the row filter to the table
spark.sql("""
  ALTER TABLE production.customers.profiles
    SET ROW FILTER production.customers.region_filter ON (customer_region);
""")

# Verify — EU analyst will only see EU rows
result = spark.sql("SELECT COUNT(*) FROM production.customers.profiles")
print(f"Rows visible to current user: {result.collect()[0][0]}")

Step 4 — Querying the Audit Log via System Tables

Unity Catalog writes all access events to system tables. You can query them directly in Databricks SQL to build compliance reports.

# Who accessed the customers table in the last 7 days?
audit_df = spark.sql("""
  SELECT
    event_time,
    user_identity.email          AS user_email,
    request_params.table_full_name AS table_accessed,
    request_params.operation     AS operation,
    response.status_code         AS status
  FROM system.access.audit
  WHERE
    event_date >= current_date() - INTERVAL 7 DAYS
    AND request_params.table_full_name LIKE 'production.customers.%'
  ORDER BY event_time DESC
""")

audit_df.show(20, truncate=False)

# Summarise by user
audit_df.groupBy("user_email", "operation") \
    .count() \
    .orderBy("count", ascending=False) \
    .show()

Things to Watch in Production

One metastore per region. Unity Catalog metastores are regional. If you have workspaces in East US and West Europe, you need two metastores. Cross-metastore queries are not supported so design your catalog layout with region affinity in mind.

External vs managed tables. Managed tables store data in the metastore's managed location — dropping the table drops the data. External tables store data in your own ADLS path — dropping the table leaves the data. For production datasets, external tables are safer since they survive accidental DROP TABLE.

Group sync from Azure AD. Don't manage Unity Catalog groups manually. Set up SCIM provisioning from Azure AD so group membership is always in sync with your identity provider. Manual groups drift and create access control gaps.

Column masks apply at query time not write time. This means masking doesn't protect data in the underlying Delta files — only in query results. If someone has direct storage access to the ADLS container, masks don't help. Lock down storage-level access separately via ADLS ACLs.

Wrapping Up

Databricks Unity Catalog solves the governance problem that every data platform eventually hits: you can't scale access control across teams and workspaces with manual Hive metastore permissions and spreadsheets. The combination of the three-level namespace, column masking, row-level security, and system table audit logs gives you a governance model that satisfies both engineers and compliance teams.

The best time to adopt Unity Catalog is before your data platform gets complicated. The second best time is now.

References

• Databricks Unity Catalog Overview
• Unity Catalog — Row Filters and Column Masks
• Unity Catalog System Tables
• SCIM Provisioning for Unity Catalog
• Managed vs External Tables in Unity Catalog
• Unity Catalog Privileges Reference
• Delta Sharing with Unity Catalog
• Unity Catalog Best Practices