Software Engineer having a total of 2 Years' experience in the IT industry with knowledge AWS - services and Databricks. Focus on designing and delivering the most optimum and critical business solutions for Data Engineering.
Project 1 : Vanguard.
Description: The client is a global market leader with the investments.
Role/Title: Junior Data Engineer
Responsibilities:
• Assisted in the development and implementation of the ETL process.
• Conducted data cleaning and preparation tasks.
• Collaborated with data engineers in the file-based ingestion process.
Involved in change record management tasks during production implementation.
Project 2: National Australia Bank (NAB)
Description: NAB Data Hub (NDH) is an enterprise-wide data lake stored in the cloud. NDH will enable scaled data sharing across the bank, leading to better use of data for insights, reporting, advanced analytics, and bulk data access.
Role/Title: Data Engineer.
Responsibilities: Experience in various data lake layers – ingestion, curation/transformation, and serving for batch files and stream data.
• Experience in handling data with different file formats, like .bin, .csv, .ctl, .xml, ., and Parquet.
Responsible for building JSON scripts, working with other team developers for code review, and performing unit testing.
Ingested stream and batch data to S3 as per NDH standards using Jenkins and Apache Airflow.
• Implemented data lineage, and applied reconciliation checks for data pipelines in NDH.
• Performed customizations and served data as per client’s requirements via s3 access points, IAM roles and policies, and AWS managed SFTP servers.
• Developed ETL pipelines in Redshift using SQL.
• Performed testing using Spark data frames in EMR – Pyspark
• Ingested batch files and stream data from source to target tables using HVR and Databricks.
• Involved in production releases activities and managing incident resolutions.
Above given Details are to the best of my knowledge and belief looking for an opportunity to prove my Excellency and dedicate of my services to the organization and carve out and identity for myself.