Khushal Karim

Big Data Engineer

IBM • Tiempo completo

Dec 2023 - Present • 2 yrs 5 mos

1. Extensive experience designing, building, and scaling large-scale distributed Big Data platforms and data pipelines across cloud and on-prem environments. 2. Architected and optimized high-performance ETL/ELT pipelines to process massive structured and unstructured datasets, ensuring scalability, fault tolerance, and low-latency data processing. 3. Strong expertise in distributed data processing using Hadoop, Spark, Hive, and PySpark, with advanced knowledge of data modeling, partitioning, indexing, and performance tuning for petabyte-scale workloads. 4. Designed and maintained robust batch and real-time data pipelines, implementing data lakes, data warehouses, and feature engineering workflows for large-scale analytics. 5. Proven ability to convert complex business and data requirements into reliable, production-grade Big Data architectures with strong governance, monitoring, and performance optimization practices.

Azure Data Engineer

Volmatica • Tiempo completo

Mar 2022 - Dec 2023 • 1 yr 9 mos

1. Designed and implemented end-to-end data solutions in Microsoft Fabric, including ingestion, transformation, storage, and consumption, enabling scalable and unified analytics. 2. Built and orchestrated ETL/ELT pipelines using Fabric Data Factory (Pipelines & Dataflows Gen2) and Spark workloads, integrating data from cloud, on-premises, and API sources. 3. Developed and managed Lakehouse and Delta Lake architectures on OneLake, ensuring high-performance storage, versioning, ACID compliance, and reuse across workloads. 4. Created AI- and analytics-ready datasets through Spark notebooks (PySpark/SQL/Scala), supporting machine learning pipelines, Power BI dashboards, and advanced analytics. 5. Ensured data governance, security, monitoring, and operational reliability, leveraging Fabric’s cataloging, access controls, lineage, and scheduling features for production-grade pipelines.