๐ Building a 25-Year Backfill Pipeline for the National Library of Korea API
How I Designed a Reliable, Auto-Resuming ETL to Collect Decades of Book Data โ Without Airflow 1. Why I Built This The National Library of Korea (NLK) provides a public API called Seoji โ a bibliographic catalog of all registered books in Korea. I wanted to collect the entire dataset, from January 2000 to December 2024, and store it in my PostgreSQL database (Supabase). It sounded simple at first โ just a loop over API pages. But in practice, I had to solve: ...