12 posts
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew, ...
> Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured S...
> A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized e...
> From raw data to a registered, served fine-tuned model — a production walkthrough using Databricks, Apache Spark, MLflow, and Hugging Face Transformers. --- T...
> A deep dive into the medallion architecture, Delta Lake internals, Z-ordering, and optimized Spark writes — the patterns that separate hobby projects from pro...
If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta Lake, both ...
Airflow is powerful but it comes with a tax: a server to maintain, a DAG repo to manage, a scheduler to babysit, and a support burden every time a new team memb...
Databricks bills by DBU (Databricks Unit). DBUs add up fast when you have multiple teams running notebooks, jobs, and SQL warehouses all day. The good news is t...
Azure Databricks doesn't live in isolation. In most enterprise Azure environments it sits downstream of Azure Data Factory, which orchestrates ingestion from da...
Batch pipelines are predictable. They run, they finish, you check the results. Streaming pipelines are alive — they never stop, failures compound, and a small m...
The hardest part of scaling a data platform isn't the compute. It's knowing who can access what, proving it to auditors, and not breaking everything when someon...
If you've ever spent more time debugging broken Spark jobs than actually building pipelines, Databricks Delta Live Tables (DLT) is worth your attention. Instead...