15 posts
There's an awkward moment in every agentic data project. The agent works. It writes decent SQL, it reasons about schema, it proposes a migration that looks righ...
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew, ...
Most ML teams are great at training models. Very few are great at shipping them. The gap between a notebook that works and a model that reliably serves producti...
> Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured S...
> A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized e...
> From raw data to a registered, served fine-tuned model — a production walkthrough using Databricks, Apache Spark, MLflow, and Hugging Face Transformers. --- T...
> A deep dive into the medallion architecture, Delta Lake internals, Z-ordering, and optimized Spark writes — the patterns that separate hobby projects from pro...
If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta Lake, both ...
Airflow is powerful but it comes with a tax: a server to maintain, a DAG repo to manage, a scheduler to babysit, and a support burden every time a new team memb...
Databricks bills by DBU (Databricks Unit). DBUs add up fast when you have multiple teams running notebooks, jobs, and SQL warehouses all day. The good news is t...
The default Azure Databricks workspace is more open than most security teams are comfortable with. Control plane traffic goes over the public internet, secrets ...
Azure Databricks doesn't live in isolation. In most enterprise Azure environments it sits downstream of Azure Data Factory, which orchestrates ingestion from da...
Batch pipelines are predictable. They run, they finish, you check the results. Streaming pipelines are alive — they never stop, failures compound, and a small m...
The hardest part of scaling a data platform isn't the compute. It's knowing who can access what, proving it to auditors, and not breaking everything when someon...
If you've ever spent more time debugging broken Spark jobs than actually building pipelines, Databricks Delta Live Tables (DLT) is worth your attention. Instead...