A seasoned IT professional with 18 years of experience, specializing in Data Engineering with a focus on Azure cloud technologies. My core expertise lies in designing, developing, and deploying robust and scalable data solutions, leveraging Azure Data Factory (ADF) for orchestrating complex ETL pipelines. Proficient in building data transformations using Azure Databricks (PySpark) and optimizing data loading into various target systems. Possess deep SQL development skills, enabling me to write efficient queries and stored procedures for data manipulation and analysis. Extensive experience in ETL processes using tools like SSIS, Talend, and specifically Azure Data Factory, allowing me to handle diverse data integration scenarios. My background includes data warehousing and dimensional modeling, enabling me to design efficient data models for business intelligence. I am adept at performance tuning and optimization of data pipelines and SQL queries.
Orchestrated pipelines to extract data from source systems, transform it using PySpark and Spark SQL, and load it into destination databases. Implemented intricate business logic within the data transformation processes. Utilized control tables to govern pipeline. execution. Integrated the pipelines seamlessly to support higher-level integration in the Medallion Architecture, serving. as a critical component for reporting in Power BI for gold-layer applications.
Technologies Used: Data bricks Unity catalog, Python, Kafka, Event Hub, Pyspark,,Blob storage,Synapse,Spark SQL, and Airflow
• Integrated Kafka and Azure Event Hub for real-time streaming data ingestion into Databricks, ensuring seamless data flow from multiple source systems into the data pipelines.
• Responsible in maintain the security on blob storage and provide required access to azure services.
• Designed an all-encompassing table for the purpose of logging into all ETL processes and their respective current. execution timestamps.
• Utilized Kafka's event-driven architecture to capture and process high volumes of streaming data, while leveraging Azure Event Hub to manage and orchestrate event streams.
• have worked extensively on Azure Synapse to manage complex data loads, orchestrate ETL workflows also utilized YAML for configuration management to automate deployment processes.
• Installed, configured, and maintained SQL Server 2017 databases, ensuring high availability and performance.
• Performed routine database maintenance, including patching, backups (full, differential, and transactional), restores, upgrades, and configuration adjustments.
• Developed and implemented comprehensive backup and recovery strategies to minimize data loss and downtime.
• Proactively monitored database performance using performance monitor, extended events, and other tools to identify and resolve bottlenecks.
• Collected and analyzed database statistics, query plans, and other performance metrics to identify areas for improvement and recommend optimizations.
• Optimized database performance by tuning queries, indexes, stored procedures, and server configurations.
• Implemented and maintained database security measures, including user permissions, roles, and auditing.
• Troubleshot and resolved database issues, minimizing downtime and ensuring business continuity.
• Automated routine database tasks using scripting languages like T-SQL and PowerShell.
• Collaborated with developers and other stakeholders to design and implement efficient database solutions.
• Supported Monster.com’s Job board applications which was designed on ASP.Ne, Asp, SQL Server
• Worked as Trainee .Net developer and involved in designing various internal portals of company