Events data warehouse
Endeavor OnLocation
This project automated data extraction from Optimo through REST APIs, transforming it into flattened CSV files optimised for loading into Snowflake. The result was a dependable data foundation for analytics, reporting and data-driven decisions.
What I delivered
Impact,
point by point.
- 01
Developed and optimised Python scripts for extraction, transformation and flattening.
- 02
Built automated ETL pipelines for timely, accurate data preparation and loading.
- 03
Implemented error handling and logging to monitor and troubleshoot every extraction.
- 04
Integrated data validation to protect integrity before loading into Snowflake.
- 05
Designed the solution to scale with growing data volumes.
- 06
Worked closely with data engineers and analysts to align with business analytics needs.
Technology
Built with
- Python
- Snowflake
- Apache Spark
- Pandas
- AWS ECS
- AWS Lambda
- S3
- Webhooks