Data Engineering Skills

Data Engineer skills I'm Familiar with

Python(Programming Language) - Used Python for building ETL pipelines, automating data workflows, and integrating ML models with tools like Spark, Scikit-learn, and TensorFlow across AWS and Azure platforms.

SQL - Used SQL extensively to query, transform, and analyze structured data across platforms like Azure Synapse, AWS Redshift, and PostgreSQL. Played a key role in building efficient ETL workflows, optimizing data models, and enabling real-time analytics and reporting.

Microsoft Azure - Designed and deployed end-to-end data solutions using Azure services, including Data Factory, Synapse, and Event Hubs. Streamlined ETL workflows, enabled real-time data processing, and implemented secure, compliant pipelines using Azure DevOps and Key Vault—enhancing system reliability, scalability, and performance.

AWS - AWS provided the scalable cloud infrastructure needed to process our high-volume health insurance data while maintaining strict security standards for sensitive medical information. Its integrated services like Glue, Kinesis, and Lambda allowed us to build end-to-end automated workflows that significantly reduced processing times and increased operational efficiency.

Apache Kafka - Apache Kafka served as the backbone of our real-time data streaming architecture, enabling the processing of over 100,000 financial record daily with minimal latency. Its distributed messaging system allowed us to build fault-tolerant pipelines that ensured critical insurance and loan data flowed seamlessly between systems while maintaining data integrity during peak processing periods.

Snowflake - Snowflake's cloud data warehouse platform transformed our financial data analytics capabilities by providing elastic scalability that effortlessly handled our growing volumes of insurance claims and transaction data. Its unique architecture allowed us to implement secure data sharing between departments while maintaining strict governance controls, enabling our analysts to perform complex risk modeling and claims pattern analysis without compromising on performance or data security.

Apache Airflow - We implemented Apache Airflow to orchestrate complex ETL workflows across our financial data ecosystem, which significantly improved process reliability and observability. Its scheduling capabilities and comprehensive monitoring dashboards allowed our team to efficiently manage dependencies between hundreds of daily data tasks while providing clear visibility into processing bottlenecks and failure points.

DevOps & CI/CD - Docker, Kubernetes, Terraform, Jenkins, Git, Gitlab

Data Analysis & Visualization - PowerBI, Tableau, Grafana, Excel

Machine Learning & Automation - Scikit-learn, Tensorflow, PyTorch