As a data engineer, there’s nothing quite like the satisfaction of reaching the “final hurdle” in a complex distributed system setup. Today, I want to share a frustrating but very common issue with Apache Spark Structured Streaming + Kafka: 👉 the dreaded Failed to find data source: kafka error. 🧨 The Problem: Everything Seems Right — But Kafka Won’t Load Picture this: Spark cluster is up and running. Postgres connection works. Kafka is producing events. Your code calls .readStream.format("kafka")… And then: ...

Why I Run dbt Inside Airflow Docker Container In modern data engineering pipelines, dbt and Airflow often work side by side. One common design decision is how to run dbt alongside Airflow: Should dbt run in its own container, orchestrated via API or CLI call? Or should dbt run directly inside Airflow’s Docker container as part of the DAG? After experimenting with both, I prefer running dbt inside Airflow’s Docker container. ...

In real-world data engineering pipelines, one of the most common mistakes is postponing data cleansing until too late in the pipeline. The cleaner your upstream data is, the simpler and more maintainable your downstream models will be. Let’s break it down. ✅ The Principle Whenever possible, cleanse your data as early as possible — ideally at the staging layer. ✅ The Why 1️⃣ Clear Separation of Responsibilities Staging models are responsible for: ...

😩 The Problem Setting up Jekyll with Docker sounds easy, but I ran into: platform issues (arm64 vs amd64) - I use Apple Silicon Macbook (M1) bundle install headaches Since I was building this for my personal GitHub Pages site, I also had to make sure it stays compatible with GitHub Pages gem versions while being easy to develop locally. 🛠 My Clean Solution I ended up building this Docker setup. It works for me at last. ...

While setting up Airflow + dbt projects with Docker, you may run into this common error message and its solutions. 🔍 Problem 1: Platform Architecture Mismatch Error message: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) My Mac is running on ARM (Apple Silicon - M1/M2/M3). The official dbt Docker image is built for amd64 (x86-based). As a result, Docker tries to run cross-architecture using QEMU emulation, which sometimes leads to internal Python path issues → surfaces as the dbt dbt --version error. This is not a simple dbt bug — the root cause is platform mismatch. ...

Common issues you will often encounter when running Airflow with Docker. ❗ Issue 1 — .env file is not visible inside Airflow container 🔍 Symptom Summary The .env file exists at the project root. But inside the Airflow container, load_dotenv() fails to read it. The reason: Docker automatically passes .env as environment variables. But Docker does not copy or mount the file itself into the container. Therefore, load_dotenv() has no file to read. ✅ Solution 1️⃣ Add volume mount for .env in docker-compose.yml This way, the .env file becomes available inside the container at the correct path. ...

If you start working with Airflow a bit more seriously, you’ll quickly notice that it’s usually split into multiple services: airflow-init airflow-scheduler airflow-webserver At first, you may wonder: “Why do we need to split them up like this?” Well — this is actually the standard production architecture. Let’s break it down in simple, practical terms. 1️⃣ airflow-init — Preparation Step Also sometimes called airflow-db-migrate or airflow-bootstrap. This runs only once when you initialize Airflow. ...

✨ Try It 👉 Launch the Potting Soil Calculator 📬 Source Code GitHub: https://github.com/namikimlab/potting-soil-calculator 🪴 Why I Built This When planting in pots, figuring out how much soil you need can be surprisingly tricky. The volume depends on the pot’s shape, size, height, and quantity — and beginner gardeners often don’t have a clear way to calculate it. So I decided to create a tool that allows users to quickly and intuitively calculate the soil volume needed for repotting. ...

Role dbt Does Well Python Does Better Structured data cleaning (staging) ✅ Possible, but inconvenient Designing mart table structures ✅ Also possible User-specific calculations ❌ Inconvenient ✅ Super flexible Scoring, conditional matching, if-else logic ❌ Very cumbersome ✅ Ideal Filtering based on user input ❌ Not possible ✅ Core feature Explaining recommendations, tuning logic ❌ ✅ Fully customizable For Example -- This kind of logic is painful in dbt... SELECT CASE WHEN user.age BETWEEN policy.min_age AND policy.max_age THEN 30 ELSE 0 END + CASE WHEN user.income < policy.income_ceiling THE_ ELSE 0 END + ... In dbt, the concept of a “user” doesn’t even exist dbt is built for models that apply the same logic to everyone Python, on the other hand, can generate different recommendations per user based on input 👉 dbt is great for static modeling, but dynamic, user-input-driven recommender systems are better suited for Python. ...

✨ Project Summary Assuming I am working for a fintech company, I built a batch pipeline that automatically aggregates → transforms → analyzes credit card data. Since I couldn’t use real data, I used synthetic transaction data generated using Faker, but I believe it was sufficient for the purpose of designing the overall data flow and structure. 🎯 Goal “Build an Airflow pipeline that processes realistic financial data with Spark, analyzes and stores them.” ...