๐ Building a Batch Data Pipeline with AWS, Airflow, and Spark
โจ Project Summary Assuming I am working for a fintech company, I built a batch pipeline that automatically aggregates โ transforms โ analyzes credit card data. Since I couldnโt use real data, I used synthetic transaction data generated using Faker, but I believe it was sufficient for the purpose of designing the overall data flow and structure. ๐ฏ Goal โBuild an Airflow pipeline that processes realistic financial data with Spark, analyzes and stores them.โ ...