MLOps: Building a CI/CD Pipeline for ML Models on Azure Databricks
9 min read4.7k
Most ML teams are great at training models. Very few are great at shipping them. The gap between a notebook that works and a model that reliably serves producti...
#Databricks
13 posts
Real-Time AI Feature Engineering with Spark Structured Streaming and Databricks Feature Store
11 min read5k
> Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured S...
Apache Spark Query Optimization on Databricks: Catalyst, AQE, and Photon Engine
13 min read4.6k
> A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized e...
Fine-Tuning LLMs at Scale with Databricks MLflow and Spark
11 min read4.7k
> From raw data to a registered, served fine-tuned model — a production walkthrough using Databricks, Apache Spark, MLflow, and Hugging Face Transformers. --- T...
Building Production-Grade Delta Lake Pipelines with Apache Spark on Databricks
9 min read4.7k
> A deep dive into the medallion architecture, Delta Lake internals, Z-ordering, and optimized Spark writes — the patterns that separate hobby projects from pro...
Azure Databricks vs Microsoft Fabric: An Honest Guide to When to Use What
7 min read3.3k
If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta Lake, both ...
Databricks Workflows: Orchestrating Multi-Task ML Pipelines Without the Airflow Overhead
7 min read5.4k
Airflow is powerful but it comes with a tax: a server to maintain, a DAG repo to manage, a scheduler to babysit, and a support burden every time a new team memb...
Azure Databricks Cost Optimization: Autoscaling, Spot Instances, Photon, and Cluster Policies That Actually Save Money
7 min read4.5k
Databricks bills by DBU (Databricks Unit). DBUs add up fast when you have multiple teams running notebooks, jobs, and SQL warehouses all day. The good news is t...
Azure Databricks Security: Private Link, Azure AD, and Key Vault Done Right
8 min read3.5k
The default Azure Databricks workspace is more open than most security teams are comfortable with. Control plane traffic goes over the public internet, secrets ...
Azure Databricks + Azure Data Factory and ADLS Gen2: Building a Production Data Integration Pipeline
7 min read4.5k
Azure Databricks doesn't live in isolation. In most enterprise Azure environments it sits downstream of Azure Data Factory, which orchestrates ingestion from da...
Databricks Spark Structured Streaming: Building Real-Time Pipelines That Actually Work in Production
6 min read4.7k
Batch pipelines are predictable. They run, they finish, you check the results. Streaming pipelines are alive — they never stop, failures compound, and a small m...
Databricks Unity Catalog: Data Governance and Access Control at Scale
7 min read4.4k
The hardest part of scaling a data platform isn't the compute. It's knowing who can access what, proving it to auditors, and not breaking everything when someon...
Databricks Delta Live Tables: Declarative Pipeline Orchestration Made Simple
6 min read5.1k
If you've ever spent more time debugging broken Spark jobs than actually building pipelines, Databricks Delta Live Tables (DLT) is worth your attention. Instead...