MG Krishna Reddy Naredla

D+ Plan Switch Funnel
Built a Snowflake data mart to track user plan-switching behavior on Disney+; used PySpark and Airflow to develop scalable ETL pipelines, with CI/CD managed via GitHub Actions.

ESPN Sign-Up Funnel
Developed ESPN sign-up funnel tracking using PySpark and Snowflake; automated data pipelines with Airflow and deployed via GitHub Actions to support product analytics. Funnel : Worked on developing on sign up funnel for espn

• Entity Resolution: collaborated within a team to perform entity resolution using Apache Spark for datasets up to 20TB, managed data processing on Google Cloud Dataproc, orchestrated workflows with Google Cloud Composer, implemented CI/CD pipelines through Git Actions.

• Declarative Metrics: customized fork of MetricFlow, incorporating tailored components for CI/CD, version control of metrics, detection of data drift across various versions, and the ability to access data from multiple sources through APIs and a Python library.
• Data Quality Framework: An automated data control framework built upon AWS Deequ, designed to ensure data quality in Spark pipelines without the need for manual coding.
• Ad-hoc analysis: python library to perform ad hoc analysis on hcc (Hierarchical Condition Categories)

• Data Streaming: Requirement was to stream data from IBM Mq into Hive and also process each record through rest service to another system. Implemented end to end solution. Created a custom spark receiver to consume data from IBM MQ into spark streaming. Data volume used to be around 5k to 7k messages per 5 minutes.
• Data Ingestion Framework: Developed a data ingestion framework using PySpark, initially for Hive, handling huge data volumes (10GB to 50GB). In its upgraded version, I introduced schema management and Delta Lake support, with configurable options for each table. It's platform-agnostic and built for Kubernetes deployment.
• Data pipeline: Argo workflow is used as an orchestration tool on k8s for spark jobs, since argo is mainly a yaml based created a python wrapper to build pipelines from python code. link: https://github.com/IndustrialDataOps/argoflow.
• ML Model version: python helper library to version scikit based ml models, store the artifacts on s3 and to deploy them based on version.