Atiyah Rehman
Woodstock,ON
Summary
Seasoned Data Engineer with 13 years of expertise in developing and optimizing robust data pipelines and systems. Proficient in ETL processes, data warehousing, and database management, leveraging advanced skills in Python, SQL, and Java. Extensive experience with cloud platforms including Azure and AWS, adept at architecting and implementing scalable solutions for data storage, processing, and analytics. Demonstrated success in collaborating across teams to deliver actionable insights that drive strategic decision-making. Strong problem-solving abilities and effective communication skills.
Overview
13
13
years of professional experience
Work History
Sr. Big Data Engineer / Hadoop Developer
Morgan Stanley
New York City Metropolitan Area
08.2021 - Current
- Collaborated with managers and stakeholders to understand core business requirements
- Implemented a generic ETL framework with high availability for bringing related data for Hadoop Cassandra from various sources using spark
- Imported data from various sources into HDFS using Sqoop, applied transformations using Hive, Apache Pyspark, and loaded data into Hive tables or AWS S3 buckets
- Transforming business problems into Big Data solutions and define Big Data strategy and Roadmap
- Worked extensively with AWS services such as S3, Redshift, Glue, Lambda, Athena, and CloudTrail.
Big Data Engineer / Hadoop Developer
JP Morgan
NY
08.2024 - 07.2024
- Worked independently on understanding the business needs and goals of the organizations
- Developed Data Integrity and Data Quality components like DBDataQualityChecks, File Data Integrity checks, Balance Comparison checks for the incoming binary files and as well as certain control points in the ETL
- Developed Map Reduce Program for Generating Unique key for every incoming new record (Universal Key Generator)
- Developed UDF’s for hive and pig to support extra functionality provided by Teradata
- Worked on Avro and Parquet File Formats with snappy compression.
Big Data Engineer / Hadoop Developer
Marlin Capital Solutions
NJ
12.2024 - 07.2024
- Worked independently on understanding the business needs and goals of the organizations
- Utilized data analytics to identify trends and areas for improvement in user engagement at GHI SAAS company, leading to a 15% increase in retention rates
- Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and extracted the data from Oracle into HDFS using Sqoop.
Big Data Engineer
Vanguard
Malvern, PA
08.2017 - 11.2018
- Worked with business analysts, business stakeholders, SMEs to analyze business requirements
- Migrating data from multiple source systems to Hadoop distributed file systems for data analysis
- Building pipelines for preprocessing and data cleaning using oozie
- Installed Oozie workflow engine to run multiple Hive and Pig Jobs
- Created hive managed and external tables
- Designed AWS Glue pipelines to ingest, process, and store data interacting with different services in AWS
- Populated HDFS and Cassandra with huge amounts of data using Apache Kafka.
Data Engineer (Hadoop Developer)
Quicken Loans
MI
11.2015 - 07.2017
- Using the components Ec2,S3,SWS,SQS,RDS,RedShift,EMR in day-to-day activities
- Analyzing the data of source systems and mapping the data to our target WMS which contains 9 consolidated WMS tables
- Developed a python script to transfer data from on-premises to AWS S3
- Validated the Map reduce, Pig, Hive Scripts by pulling the data from the Hadoop and validating it with the data in the files and reports
- Developed a python script to hit REST API’s and extract data to AWS S3.
Data Engineer
The Janssen Pharmaceutical Companies of Johnson & Johnson
NJ
11.2013 - 10.2015
- Using the components Ec2,S3,SWS,SQS,RDS,RedShift,EMR in day-to-day activities
- Installed/Configured/Maintained Apache Hadoop clusters for application development and Hadoop tools like Hive, Pig, HBase, Flume, Oozie Zookeeper and Sqoop
- Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs
- Expert in creating Hive UDFs using Java to analyze the data efficiently
- Wrote MapReduce jobs using Java API and Pig Latin
- Wrote Pig scripts to run ETL jobs on the data in HDFS and further do testing.
Spring MVC/Core Java Developer
American Water
Cherry Hills, NJ
12.2011 - 10.2013
- Collaborated with cross-functional teams and stakeholders to gather business requirements
- Involved in requirement analysis and played a key role in project planning
- Successfully completed the Architecture, Detailed Design & Development of modules
- Interacted with end users to gather, analyze, and implement the project
- Designed and developed web components and business modules through all tiers from presentation to persistence
- Used hibernate for mapping from Java classes to database tables
- Developed the Action Classes, Action Form Classes, created JSPs using Struts tag libraries and configured in Struts-config.xml, Web.xml files.
Education
Master of Science - Computer Science
University of Houston Clear Lake
Houston Texas USAB.Tech - Computer Science
Jawaharlal Nehru Technological University
Skills
- ETL/ELT Development
- Data Engineering
- Data Warehouse
- Data Visualization
- Data Analysis
- Data Lake & Data Modeling
- Cloud Big Data
- Distributed Data Architecture
- Requirement Gathering
- Design & Deliver Solution
- Data Management
- Performance Optimization
- Team Management
- Client Relationship Management
Timeline
Big Data Engineer / Hadoop Developer
Marlin Capital Solutions
12.2024 - 07.2024
Big Data Engineer / Hadoop Developer
JP Morgan
08.2024 - 07.2024
Sr. Big Data Engineer / Hadoop Developer
Morgan Stanley
08.2021 - Current
Big Data Engineer
Vanguard
08.2017 - 11.2018
Data Engineer (Hadoop Developer)
Quicken Loans
11.2015 - 07.2017
Data Engineer
The Janssen Pharmaceutical Companies of Johnson & Johnson
11.2013 - 10.2015
Spring MVC/Core Java Developer
American Water
12.2011 - 10.2013
Master of Science - Computer Science
University of Houston Clear Lake
B.Tech - Computer Science
Jawaharlal Nehru Technological University
Similar Profiles
Bernard E. FiddlerBernard E. Fiddler
Executive Director at Morgan StanleyExecutive Director at Morgan Stanley
Kristin A. AlexanderKristin A. Alexander
Vice President Group Director at Morgan Stanley, Wealth ManagementVice President Group Director at Morgan Stanley, Wealth Management
Zsuzsa Máthé-MülbacherZsuzsa Máthé-Mülbacher
Fixed Asset Team TL at Morgan StanleyFixed Asset Team TL at Morgan Stanley
Alexander FinanceAlexander Finance
Wealth Management Intern at Morgan StanleyWealth Management Intern at Morgan Stanley
MATHEW MACPHERSONMATHEW MACPHERSON
Apprentice Electrician at Industrial Contracting SolutionsApprentice Electrician at Industrial Contracting Solutions