Ab Ovo - Sunnyvale, CA
posted 2 months ago
As a Data Engineer, you will play a crucial role in building and maintaining robust, scalable data pipelines that support Walmart Connect and its advertising needs. This position requires a strong background in software development and distributed systems, with a focus on data engineering concepts, database design, ETL processes, and data mining. You will be responsible for working with various data technologies, including SQL, Python, Spark, Scala, and Hadoop, to ensure efficient data processing and analytics. Your expertise in ETL tools such as Apache Airflow, Digdag, and Oozie will be essential in managing data workflows effectively. In this role, you will also work on cloud platforms like Azure and Google Cloud for data storage, processing, and analytics. You will create and deploy large-scale, containerized applications using Docker and Kubernetes in public clouds, ensuring that the applications are scalable and reliable. Managing data engineering projects from design to deployment will be a key responsibility, and you will need to ensure timely delivery while meeting project goals. As a self-starter and mentor, you will lead engineering projects and coordinate technical design discussions to drive the technical architecture of the systems you work on. Your role will involve navigating complex systems and delivering production-ready code that adheres to operational best practices, including monitoring, alerting, validation, and exception handling. You will support data operations, including on-call rotations to troubleshoot production issues, and collaborate with cross-functional tech teams to identify performance bottlenecks and areas for optimization. Excellent communication skills are essential for effectively collaborating with various teams and presenting performance analysis findings.