Data Engineer

Ab Ovo - Sunnyvale, CA

Full-time - Mid Level

Sunnyvale, CA

Professional, Scientific, and Technical Services

About the position

As a Data Engineer, you will play a crucial role in building and maintaining robust, scalable data pipelines that support Walmart Connect and its advertising needs. This position requires a strong background in software development and distributed systems, with a focus on data engineering concepts, database design, ETL processes, and data mining. You will be responsible for working with various data technologies, including SQL, Python, Spark, Scala, and Hadoop, to ensure efficient data processing and analytics. Your expertise in ETL tools such as Apache Airflow, Digdag, and Oozie will be essential in managing data workflows effectively. In this role, you will also work on cloud platforms like Azure and Google Cloud for data storage, processing, and analytics. You will create and deploy large-scale, containerized applications using Docker and Kubernetes in public clouds, ensuring that the applications are scalable and reliable. Managing data engineering projects from design to deployment will be a key responsibility, and you will need to ensure timely delivery while meeting project goals. As a self-starter and mentor, you will lead engineering projects and coordinate technical design discussions to drive the technical architecture of the systems you work on. Your role will involve navigating complex systems and delivering production-ready code that adheres to operational best practices, including monitoring, alerting, validation, and exception handling. You will support data operations, including on-call rotations to troubleshoot production issues, and collaborate with cross-functional tech teams to identify performance bottlenecks and areas for optimization. Excellent communication skills are essential for effectively collaborating with various teams and presenting performance analysis findings.

Responsibilities

• Build and maintain robust, scalable data pipelines for ingesting, transforming, and storing large volumes of data to support Walmart Connect and its advertising needs.
• Work on cloud platforms like Azure and Google Cloud for data storage, processing, and analytics.
• Create and deploy large-scale, containerized applications using Docker and Kubernetes in public clouds like Google Cloud Platform and Microsoft Azure.
• Manage data engineering projects from design to deployment, ensuring timely delivery and meeting project goals.
• Self-starter, mentor who can lead engineers and engineering projects.
• Coordinate, create, and complete technical design discussions to drive technical architecture.
• Navigate complex systems and deliver highly scalable and reliable production-ready code.
• Support data operations including on-call rotations to troubleshoot production issues partnering with cross-functional teams.
• Collaborate with cross-functional tech teams including developers, architects, operations to identify performance bottlenecks, system issues, and areas for optimization.

Requirements

• Bachelor's or master's degree in computer science, Software Engineering, or a related field.
• Proven experience (7+ years) as a Data Engineer, preferably with a focus on software development and distributed systems.
• Solid understanding of data engineering concepts, database design, ETL processes, and data mining.
• Proficiency in working with data technologies including SQL, Python, Spark, Scala, Hadoop, and related technologies and tools.
• Experience with ETL tools such as Apache Airflow, Digdag, and Oozie.
• Experience with CI/CD processes and tools such as Jenkins and Maven.
• Strong understanding of metadata management, data lineage, and data quality tools.
• Familiarity with Cloud Computing Services such as Google Cloud Platform and Microsoft Azure, as well as Distributed Storage Systems like Hive and Elastic Search is a plus.
• Experience implementing operational best practices such as monitoring, alerting, validation, and exception handling.
• Strong analytical and problem-solving skills with the ability to optimize, troubleshoot, and debug data processes and pipelines.
• Excellent communication skills to collaborate effectively with cross-functional teams and present performance analysis findings.
• Experience supporting and working with cross-functional teams and influencing them with solutions in a dynamic environment.
• Experience in AdTech and advertising measurement.

Nice-to-haves

• Knowledge of analytical tools such as Tableau or Looker is preferred.