Om P

Data Engineer

EllisDon • Tiempo completo

Nov 2022 - Jan 2026 • 3 yrs 2 mos

- Developed and optimized multi-cloud ETL/ELT pipelines across GCP and Azure, processing GBs–TBs of structured and semi-structured data daily and delivering reliable, analytics-ready datasets in BigQuery and Synapse. - Led migration of Apache Airflow workloads from GKE to Google Cloud Composer, redeploying ETL DAGs for MongoDB and REST API ingestion with CI/CD automation, reducing DevOps dependency by ˜60%. - Monitored and optimized Airflow DAG performance and BigQuery execution using Dataproc-based workload optimization, reducing infrastructure costs by ˜15–20%. - Maintained self-managed Kafka on Terraform-provisioned VMs with Dockerized UI & 100+ Debezium CDC connectors. -Supported zero-downtime migration from self-managed Kafka to Google Cloud DataStream, improving pipeline reliability, scalability, and reducing operational maintenance. - Developed Microsoft Fabric ELT pipelines to ingest Dynamics 365 enterprise data with automated retry mechanisms and alerting workflows, reducing manual intervention by ˜40%. - Implemented data quality frameworks across Kafka and DataStream ingestion pipelines with automated validation, monitoring, and alerts for stale or invalid data. - Designed and optimized BigQuery tables implementing SCD1 for static lookup tables and SCD2 for high-frequency Kafka and financial statement data to preserve historical accuracy. - Partnered with Data Architects, Product, Data Science, and DevOps teams in Agile ceremonies to deliver on the data roadmap and make data accessible and trustworthy across the organization.

ETL Developer

Digibee Techlab LLP • Tiempo completo

May 2020 - Dec 2020 • 7 mos

- Developed and optimized a data warehouse using Talend ETL workflows to ingest, validate, and cleanse 10+ CSV data sources into HDFS-based data tables using scheduled batch processing jobs. - Designed and automated 7+ Python-based jobs for data extraction, transformation, and workflow scheduling, improving operational efficiency and reducing manual processing. - Containerized applications using Docker and implemented GitLab CI/CD pipelines for automated builds, container registry management, and deployments, reducing release cycle time by approximately 30%. - Participated in peer code reviews, pull request approvals, Agile standups, and sprint planning sessions, ensuring maintainable code quality, collaborative development, data integrity, and timely delivery.