Events data warehouse

Endeavor OnLocation

This project automated data extraction from Optimo through REST APIs, transforming it into flattened CSV files optimised for loading into Snowflake. The result was a dependable data foundation for analytics, reporting and data-driven decisions.

Client
Endeavor
Domain
Events data warehouse
Core stack
Python, Snowflake, Apache Spark
Endeavor OnLocation product interface

What I delivered

Impact,
point by point.

  1. 01

    Developed and optimised Python scripts for extraction, transformation and flattening.

  2. 02

    Built automated ETL pipelines for timely, accurate data preparation and loading.

  3. 03

    Implemented error handling and logging to monitor and troubleshoot every extraction.

  4. 04

    Integrated data validation to protect integrity before loading into Snowflake.

  5. 05

    Designed the solution to scale with growing data volumes.

  6. 06

    Worked closely with data engineers and analysts to align with business analytics needs.

Technology

Built with

  • Python
  • Snowflake
  • Apache Spark
  • Pandas
  • AWS ECS
  • AWS Lambda
  • S3
  • Webhooks

Have a similar
challenge?