Vamsidhar Challa
Livermore
Summary
Senior Big Data Engineer with over 11 years of experience in developing scalable data platforms on AWS, GCP, and Azure. Proficient in Python, Scala, SQL, Snowflake, Databricks, Spark, and Airflow. Successfully led cross-functional data initiatives, established data contracts and quality standards, and designed high-performance ETL/ELT pipelines for enterprise applications. Proven ability to enhance stakeholder alignment and improve data platform reliability and engineering practices.
Overview
10
10
years of professional experience
Work History
Senior Big Data Engineer
Dropbox, Inc
11.2021 - Current
- Designed and optimized Snowflake, Databricks & Spark data models, improving data access efficiency and query speeds by 40%.
- Built and automated large-scale ETL/ELT pipelines, processing 1.5TB+ data per day using Snowflake, Databricks, Spark, Redshift, PSQL, and Fivetran.
- Led data migration projects from legacy platforms to Databricks, modernizing infrastructure and improving data availability.
- Developed role-based access control (RBAC) in Snowflake, Databricks & AWS Lake Formation, ensuring compliance with GDPR and CCPA.
- Implemented data health metrics and governance frameworks, ensuring integration with data observability tools similar to Monte Carlo & Atlan.
- Designed schema versioning strategies, data contracts, and anomaly detection models to maintain high-quality data standards.
- Established Git-based workflows, automated testing, and CI/CD pipelines (Jenkins, Terraform, GitHub Actions) for data pipeline deployments.
- Architected real-time streaming solutions using Kafka, Spark Streaming & AWS Kinesis, reducing data processing latency from minutes to real-time.
- Managed stakeholder relationships across Finance, Data Science, ML, Business Intelligence, and HR teams, ensuring data alignment with business objectives.
Big Data Engineer
Maestro Technologies, Inc
09.2019 - 11.2021
- Architected data pipelines for ETL/ELT processes in GCP (BigQuery, Dataflow, Cloud Functions).
- Optimized Snowflake, Databricks & Spark warehouse models, reducing query execution time by 50%.
- Developed Kafka-based real-time streaming pipelines, reducing data ingestion delays from 10 minutes to seconds.
- Managed data migrations from SQL Server, Oracle, and PostgreSQL to BigQuery, Snowflake, and Databricks.
- Developed AI-powered data validation & anomaly detection models, improving data accuracy and operational monitoring.
Data Engineer
Argus Information and Advisory Services, LLC
12.2015 - 09.2019
- Designed and developed SQL-based ETL pipelines, optimizing database performance through indexing, query tuning, and schema normalization.
- Implemented data warehouse solutions, improving data processing efficiency and enabling analytical reporting.
- Developed and deployed SSIS packages for automated data extraction, transformation, and loading from various sources.
- Assisted in data migration projects, transitioning legacy databases to modern cloud-based architectures.
- Developed monitoring and alerting solutions for data pipelines, ensuring high availability and reliability of data services.
Education
Master of Science -
University of Missouri
Kansas City, MO05-2015
Bachelor of Technology in Electrical & Electronics Engineering -
Koneru Lakshmaiah University
Vijayawada, India05-2013
Skills
- Programming languages: Python, SQL, Scala, JavaScript, Shell scripting
- Cloud platforms: Snowflake, Databricks, Redshift, BigQuery, Hive, Delta Lake
- Cloud services: AWS, GCP, Azure
- Big data frameworks: Apache Spark, PySpark, SparkSQL, Kafka
- ETL tools: Airflow, Fivetran, Glue
- Data architecture: Dimensional modeling, Data contracts, Automated validations
- Security and governance: Snowflake RBAC, AWS Lake Formation, GDPR compliance
- DevOps tools: Git, Jenkins, Terraform, Azure DevOps, GitHub Actions, Kubernetes, Docker
- Reporting tools: Tableau, Power BI, Looker
Accomplishments
• Defined enterprise-wide data contract and validation standards, enabling consistent ingestion and quality across Finance, Product, and Commerce engineering teams.
• Architected unified Spark and Databricks processing framework adopted by 6+ data teams, reducing compute costs by 30% and standardizing ETL patterns.
• Led multi-quarter migration of 750TB+ data from SQL Server, Oracle, and Hive to Snowflake and Databricks, establishing modernization roadmap and ensuring metric parity for Finance and Analytics orgs.
• Designed scalable ETL/ELT architectures powering 1.5TB/day processing, improving throughput by 40% and reducing pipeline failures by 70% across multiple domains.
• Built Python- and Scala-based ingestion and validation libraries reused by teams across the org, reducing code duplication by 40% and improving engineering velocity.
• Led cross-functional alignment with Finance, FP&A, Data Science, Commerce, and Platform teams to drive reporting consistency, metric definitions, and data governance practices.
• Implemented Spark and Snowflake performance frameworks (AQE, caching, pruning, cluster sizing) that improved runtime efficiency by 60% and reduced operational load for on-call engineers.
• Established CI/CD standards for data pipelines using Terraform, Jenkins, and GitHub Actions, ensuring reproducible deployments and improving release stability across teams.
• Designed real-time streaming architecture using Kafka, Spark Streaming, and Kinesis that supported sub-second ingestion and enabled new Finance and Growth use cases.
• Architected unified Spark and Databricks processing framework adopted by 6+ data teams, reducing compute costs by 30% and standardizing ETL patterns.
• Led multi-quarter migration of 750TB+ data from SQL Server, Oracle, and Hive to Snowflake and Databricks, establishing modernization roadmap and ensuring metric parity for Finance and Analytics orgs.
• Designed scalable ETL/ELT architectures powering 1.5TB/day processing, improving throughput by 40% and reducing pipeline failures by 70% across multiple domains.
• Built Python- and Scala-based ingestion and validation libraries reused by teams across the org, reducing code duplication by 40% and improving engineering velocity.
• Led cross-functional alignment with Finance, FP&A, Data Science, Commerce, and Platform teams to drive reporting consistency, metric definitions, and data governance practices.
• Implemented Spark and Snowflake performance frameworks (AQE, caching, pruning, cluster sizing) that improved runtime efficiency by 60% and reduced operational load for on-call engineers.
• Established CI/CD standards for data pipelines using Terraform, Jenkins, and GitHub Actions, ensuring reproducible deployments and improving release stability across teams.
• Designed real-time streaming architecture using Kafka, Spark Streaming, and Kinesis that supported sub-second ingestion and enabled new Finance and Growth use cases.
Personal Information
Title: Senior Big Data Engineer
Additional Information
Senior Data Engineer, Big Data Lead, Principal Data Engineer, Actively mentoring junior engineers in data engineering best practices., Passionate about optimizing data platforms for performance, security, and AI-driven insights.
Technical Skills
- Leadership & Mentorship
- Stakeholder Collaboration
- Data Strategy & Architecture
- Analytical Problem-Solving
- Communication & Documentation
- AI & ML Data Strategy
Certifications Professional Development
- Snowflake Certified Data Engineer
- Microsoft Certified Azure Solution Developer
- AWS Certified Data Analytics (In Progress)
- Google Cloud Professional Data Engineer (In Progress)
- Databricks Certified Data Engineer Associate
Timeline
Senior Big Data Engineer
Dropbox, Inc
11.2021 - Current
Big Data Engineer
Maestro Technologies, Inc
09.2019 - 11.2021
Data Engineer
Argus Information and Advisory Services, LLC
12.2015 - 09.2019
Master of Science -
University of Missouri
Bachelor of Technology in Electrical & Electronics Engineering -
Koneru Lakshmaiah University