VIVEKANAND BHAT

Highly experienced Azure Data Engineer with 13+ years of total industry experience and 5+ years of relevant Azure experience years of professional experience in developing secure, cost-efficient data solutions. Successfully designed and implemented 8+ projects through the entire development cycle, driving down storage costs by 25%, increasing customer satisfaction by 20%, streamlining integration and profiling processes by 40%, and more. Proven ability to architect automated environments for optimal data assets and resources, leveraging essential tools such as Azure Data factory, Databricks, ADLS Spark/Pyspark and Python.Databricks , ADLS Spark/Pyspark and Python.

• 13+ years of total work experience in data projects
• Strong data modelling and data architecture design skills (Building Data lake, Delta lake and Data warehouse)
• Experience with Azure Cloud platform and building medallion architecture.
• Ability to architect and integrate on-premises applications with Azure platforms, and multi-cloud integrations.
• Experience in building architecture using Azure Data Factory on Azure Synapse and Delta Lake
• Working experience on Azure Integration Services like event hub, functions etc.,
• Working experience on Azure Synapse analytical services like spark pool, serverless and dedicated SQL pool.
• Experience with Azure SQL Database, SQL Server, Azure Monitor and Application Insights
• Understanding of Azure Networking services (VNETs, Load Balancers), Security and Identity services (Azure AD, RBAC)
• Understanding of Azure ARM templates and worked with CI/CD using Azure DevOps
• Experience working directly with business clients to understand the requirements and design solutions.
• Strong verbal, written communication and influencing skills.

Reliance Standard Insurance Services, Insurance Data Integration and Warehouse Project  Feb-2020 to till date, Philadelphia, USA

In the rapidly evolving insurance industry, effective data management is crucial for staying competitive and providing excellent customer service. This project aims to streamline and modernize the data processing and analysis workflows for an insurance company. It involves Extract, Transform, Load (ETL) processes, data warehousing, and incremental loading of data from various source files such as XML, JSON, and CSV. Azure Data Factory, Azure Databricks, Azure Synapse Analytics (formerly SQL Data Warehouse), Spark, and PySpark will be the core technologies employed to achieve this transformation.

Project Components:

    Data Extraction:

• Extract data from various source files including XML, JSON, and CSV.
• Implement data connectors in Azure Data Factory to fetch data from these sources.
• Perform data profiling to understand the structure and quality of the source data.

    Data Transformation:

• Utilize Azure Databricks, an Apache Spark-based analytics platform, for data transformation and cleansing.
• Use PySpark to manipulate and clean the data as per business requirements.
• Handle data type conversions, data enrichment, and error handling.

  Data Loading:

• Create a staging area in Azure Data Lake Storage to temporarily store the processed data.
• Implement Azure Data Factory pipelines to load data into the staging area.
• Define incremental loading strategies to ensure only new and modified data is loaded.

Data Warehouse:

• Utilize Azure Synapse Analytics (SQL Data Warehouse) as the target data warehousing solution.
• Design the data warehouse schema to support reporting and analytics needs.
• Load data from the staging area into the data warehouse.

Data Quality and Validation:

• Implement data quality checks and validation rules to ensure the accuracy and consistency of data.
• Develop data monitoring and alerting mechanisms to detect and respond to data quality issues.

Automation and Scheduling:

• Schedule ETL processes to run at regular intervals using Azure Data Factory.
• Set up data orchestration and automation for hands-free data processing.

JPMC–CIG – Wires, Collections, ACH Payroll CBO Infrastructure Redesign -Migration, April 2017 to FEB 2020, New York, USA

Project 1:

The Banking Company Data Migration project involves migrating critical financial data and operations from on-premises infrastructure to the Azure cloud platform, leveraging Azure Data Factory, Azure Databricks, Azure Blob Storage, and Azure Data Lake. This migration aims to improve scalability, performance, and data management while ensuring data security and compliance.

Project Components and Workflow:

Assessment and Planning:

• The project will start with a thorough assessment of the existing on-premises data, applications, and infrastructure to understand the scope and complexity of the migration.
• A detailed migration plan will be developed, which includes identifying data dependencies, scheduling migration phases, and defining success criteria.

Data Extraction:

• Data will be extracted from on-premises data sources, such as relational databases, file systems, and applications.
• Azure Data Factory will be employed to create data pipelines for efficient data extraction, supporting various source data formats.

Data Transformation:

• Extracted data may require transformations to ensure compatibility with Azure cloud services and to improve data quality.
• Azure Databricks will be used for complex data transformation tasks, taking advantage of Spark-based processing and PySpark for data manipulation.

Data Staging and Storage:

• Extracted and transformed data will be staged in Azure Blob Storage and Azure Data Lake. Azure Blob Storage can be used for unstructured data, while Azure Data Lake is ideal for storing structured and semi-structured data.
• Data will be organized into logical structures and directories for efficient storage and retrieval.

Data Quality and Validation:

• Data quality checks and validations will be integrated into the ETL process to ensure the integrity of the migrated data.
• Data quality issues will trigger alerts and notifications for resolution.

Incremental Data Migration:

• The project will implement strategies for incremental data migration to keep the data in Azure up-to-date during the transition period.
• Change data capture (CDC) and data differencing techniques will be used to identify and migrate only the changed or new data.

Testing and Validation:

• Extensive testing, including unit testing, integration testing, and user acceptance testing, will be conducted to ensure the data migration is successful and that data in the Azure cloud platform is consistent and accurate.

Migration Execution:

• The migration will be executed in phases to minimize downtime and risks. Each phase will be closely monitored for issues and performance.
• Rollback procedures will be in place in case unexpected issues arise.

Security and Compliance:

• Data security will be a top priority. Azure services offer encryption, access controls, and auditing features to safeguard sensitive financial data.
• The project will ensure compliance with industry regulations, such as GDPR, HIPAA, and banking-specific regulations.

Project 2:

The Online Banking Data Migration (OLTP system) project is designed to facilitate the seamless transition of a banking company's online banking data from an existing system(Legacy) to a new system. This project will leverage PL/SQL as the ETL (Extract, Transform, Load) programming language to extract, transform, and load the data from the legacy online banking system to the new platform.  Took a lead on new digital platform schema designing/data modeling , entity mappings , writing and reviewing the program logic, Integrating different databases .

                                 The Online Banking Data Migration project, using PL/SQL as the ETL programming language, will enable the banking company to transition its online banking data smoothly to a new platform while maintaining data integrity, quality, and security. This migration will enhance the banking experience for customers and ensure business continuity for the organization. 

British Telecom – Quotation tool Development and Data Migration Projects , Feb 2014 to Apr 2017, Bangalore, India,.

 The British Telecom Company Quotation and Pricing Tool project is focused on the development and ongoing maintenance of a versatile tool that enables the company to generate accurate and competitive quotes and pricing for their telecommunications services. The tool will be built using a combination of Oracle SQL, PL/SQL, Unix shell scripting, and JavaScript to ensure efficient data management, calculations, and user-friendly interfaces. 

MetricStream Infotech - Compliance management Project . Data analysis for Asian Paints company - Dec-2012 to Feb 2014

 The Compliance Data Schema Design and Development project focuses on implementing a robust and compliant data schema using MetricStream as the primary compliance management platform. This project aims to design and develop a structured data schema for tracking and managing various compliance-related data elements while utilizing Oracle SQL and PL/SQL for data storage, retrieval, and management. 

Dell International - PHH Mortgage , Data Migrations, Backend developments - March 2010 to Dec 2012

 The PHH Mortgage Company Data Migration project is a critical initiative that involves migrating mortgage-related data from legacy systems to a modern platform. This migration will be accomplished using Informatica for ETL (Extract, Transform, Load) processes, SQL and PL/SQL for database management, and Unix shell scripting for data orchestration. The goal is to ensure a smooth and accurate transfer of data while maintaining data integrity, security, and compliance.