Pipeline

How I Designed a Reliable, Auto-Resuming ETL to Collect Decades of Book Data — Without Airflow 1. Why I Built This The National Library of Korea (NLK) provides a public API called Seoji — a bibliographic catalog of all registered books in Korea. I wanted to collect the entire dataset, from January 2000 to December 2024, and store it in my PostgreSQL database (Supabase). It sounded simple at first — just a loop over API pages. But in practice, I had to solve: ...

👉 Code, Portfolio, Blog, and LinkedIn 🎯 Batch Pipeline for Transaction Data Imagine: K-pop demon hunters launches a fintech startup for the fans. Now they have to deal with millions of credit card transactions every day — and they need to make sense of them. ...

In real-world data engineering pipelines, one of the most common mistakes is postponing data cleansing until too late in the pipeline. The cleaner your upstream data is, the simpler and more maintainable your downstream models will be. Let’s break it down. ✅ The Principle Whenever possible, cleanse your data as early as possible — ideally at the staging layer. ✅ The Why 1️⃣ Clear Separation of Responsibilities Staging models are responsible for: ...

If you start working with Airflow a bit more seriously, you’ll quickly notice that it’s usually split into multiple services: airflow-init airflow-scheduler airflow-webserver At first, you may wonder: “Why do we need to split them up like this?” Well — this is actually the standard production architecture. Let’s break it down in simple, practical terms. 1️⃣ airflow-init — Preparation Step Also sometimes called airflow-db-migrate or airflow-bootstrap. This runs only once when you initialize Airflow. ...

✨ Project Summary Assuming I am working for a fintech company, I built a batch pipeline that automatically aggregates → transforms → analyzes credit card data. Since I couldn’t use real data, I used synthetic transaction data generated using Faker, but I believe it was sufficient for the purpose of designing the overall data flow and structure. 🎯 Goal “Build an Airflow pipeline that processes realistic financial data with Spark, analyzes and stores them.” ...