Rajneesh Yadav
Toronto,Canada
Summary
Focused Big Data Engineering Consultant delivers consistent and professional work for every assignment. Offers 10+ years in Information Technology industry environments and top-notch abilities in Data Engineering skills in Healthcare, Banking & Financial domains. Reliable candidate ready to take on challenges using problem-solving and task prioritization skills to help teams succeed.
Overview
10
10
years of professional experience
Work History
AWS Data Engineer (Tech Lead)
Tata Consultancy Services
05.2024 - Current
- Company Overview: BMO
- Build data model and create data pipeline using AWS Glue, Lambda & AWS S3
- Monitor and troubleshoot the existing ETL pipelines of other use cases
- Delivered end-to-end ingestion framework using Glue and event-based lambda triggers
- Data Quality checks for file ingestion framework in AWS Glue using Python & PySpark
- Developed end-to-end solution to bring data from S3->L1 tables-> L2 and consumption layer in AWS Redshift
- Developed SQL logic as per mapping and business requirement
- Configured AWS services using proper IAM role to access S3, Glue, Redshift, lambda
- Implemented slowly changing dimension type 2 & 3 logic for tables loaded in redshift
- Worked on AWS Lake formation for medallion architecture
- Worked on AWS Athena and Azure Synapse Analytics to pull data from AWS S3 for analytical purpose
- Experience in Databricks and Jupiter lab for distributed data processing by leveraging PySpark
- Expertise in code deployment from Dev to QA, SIT & UAT using Github (CICD pipeline)
- Optimized data pipeline by writing the files in columnar format(parquet), resulting in a 35% increase in data processing speed to Athena
- Working experience in Agile methodology project cycle
- Created a SNS- integrated Lambda function that can be triggered from AWS glue jobs, providing a unified alerting mechanism for job status updates and failures
- Created a lambda function to create DDL's across different regions
Data Engineer Consultant (Tech Lead)
Infosys Ltd.
08.2018 - 04.2024
- Company Overview: CVS Pharmacy
- Design and implemented ETL pipelines using AWS glue, PySpark, and Python to process and transform clinical and pharmacy data from multiple sources, improving data accessibility and quality
- Developed PySpark based distributed data processing frameworks to handle large-scale clinical datasets efficiently, optimizing performance for AWS EMR & Glue jobs
- Built optimized Amazon Redshift data warehouses to store and analyze large-scale clinical datasets, enabling advanced analytics and reporting
- Designed and managed AWS Lake formation-based Data Lake to centralize structured and unstructured healthcare data, improving data governance and security compliance (HIPAA)
- Automated data ingestion pipelines using AWS Glue, AWS EMR, Apache Airflow, and AWS Kinesis to process streaming pharmacy data, enhancing real-time decision making for clinical interventions
- Created AIrflow DAGs to orchestrate and monitor data pipeline execution, improving workflow automation and reliability
- Designed and optimized Oracle PL/SQL procedures and complex queries for data transformation and reporting use cases
- Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
- Tuned Spark and SQL query performance by optimizing partitions, indexing, and query execution plans reducing processing time by 40%
- Implemented IAM policies, KMS encryption, and AWS Secrets Manager to secure sensitive patient and prescription data, ensuring compliance with HIPAA regulations
IIB Developer (Technology Analyst)
Infosys Ltd.
01.2015 - 07.2018
- Company Overview: American Express
- Designed and developed message flows using IBM Integration Bus (IIB) to facilitate seamless data exchange across enterprise applications
- Developed and deployed ESQL transformations and message routing for efficient data processing in IIB
- Created and Managed MQ-based message flows for real-time integration between legacy and modern systems
- Implemented SOAP and RESTful web services in IIB for external system integration
- Developed reusable message sets, message models, and DFDL schema for structured data transformation
- Implemented OAuth and TLS encryption for secure message transmission across enterprise applications
- Provided production support, debugging failed transactions, and implementing fixes to minimize disruptions
Education
B.Tech -
SRM University
05.2014
Skills
- Pyspark
- Python
- SQL
- Data Modeling
- Big Data
- ETL
- AWS
- Glue
- Redshift
- S3
- Lambda
- Athena
- AWS CDK
- Apache AirFlow
- Oracle, MySQL
Personal Information
Title: CLOUD DATA ENGINEER (AWS)
Languages
English
Full Professional
Timeline
AWS Data Engineer (Tech Lead)
Tata Consultancy Services
05.2024 - Current
Data Engineer Consultant (Tech Lead)
Infosys Ltd.
08.2018 - 04.2024
IIB Developer (Technology Analyst)
Infosys Ltd.
01.2015 - 07.2018
B.Tech -
SRM University
Similar Profiles
Vrunda ChindhadeVrunda Chindhade
HR Manager at Tata Consultancy Services, TCS, Tata Consultancy Services LimitedHR Manager at Tata Consultancy Services, TCS, Tata Consultancy Services Limited
Sreya BasuSreya Basu
Procare Analyst at Tata Steel(Tata Consultancy Services)Procare Analyst at Tata Steel(Tata Consultancy Services)
Arun VArun V
Program Manager at Tata consultancy services | Financial Ombudsman ServicesProgram Manager at Tata consultancy services | Financial Ombudsman Services
Naimish AgnihotriNaimish Agnihotri
Assistant Delivery Manager at Tata Consultancy Services, Global Shared ServicesAssistant Delivery Manager at Tata Consultancy Services, Global Shared Services
Jenny Cheng LYUJenny Cheng LYU
Salad Prep Cook (Cold Station) at Cafe on the SquareSalad Prep Cook (Cold Station) at Cafe on the Square