Senior Data Engineer

Disney Entertainment & ESPN Technology - Santa Monica, CA

Full-time - Mid Level

Santa Monica, CA

About the position

At Disney Entertainment & ESPN Technology, we are dedicated to reimagining the viewing experience for our audience while transforming Disney's media business for the future. Our Search Machine Learning (ML) team plays a crucial role in powering the ML aspects of search engines for Disney+ and Hulu platforms. This position involves working in a highly collaborative environment where ML-based services are integrated throughout the entire search journey, from query understanding to semantic retrieval and engagement-based re-ranking of content. We are committed to innovation and constantly testing new ideas to enhance user experience. In this role, you will partner closely with ML engineers and data scientists to automate and manage their data needs for regular training and online inference. You will take ownership of the entire lifecycle of data pipelines, ensuring efficient communication with upstream data teams. Your expertise in data engineering will be vital in bringing your ideas to life by engineering efficient data pipelines and supporting the feature requirements of ML engineers and data scientists. The responsibilities include designing and developing offline (batch) and near-line (streaming) data jobs, collaborating with ML and data practitioners to automate their pipelines, and building tooling and low-latency services to support event-driven architectures. You will work on multi-faceted projects with engineers from diverse backgrounds and skills, all within an Agile environment that emphasizes collaboration and teamwork.

Responsibilities

• Design and develop offline (batch) and near-line (streaming) data jobs.
• Collaborate with ML and data practitioners to automate their pipelines.
• Build tooling and low-latency services to enable and support event-driven architectures.
• Work on multi-faceted projects with engineers from diverse backgrounds and skills across teams.
• Participate in an Agile environment that focuses on collaboration and teamwork.

Requirements

• 5+ years of software experience, with 3+ years of relevant data and software experience.
• Knowledge of the Python/Java data ecosystem.
• Experience in building large datasets and scalable services.
• Experience deploying and running services in AWS, and engineering big-data solutions using technologies like Databricks, EMR, S3, Spark.
• Experience loading and querying cloud-hosted databases such as Redshift and Snowflake.
• Experience designing and developing backend microservices for large scale distributed systems using gRPC or REST.
• Experience with large-scale distributed data processing systems, cloud infrastructure such as AWS or GCP.

Nice-to-haves

• Knowledge in optimizing Spark jobs.
• Experience building streaming pipelines using Kafka, Spark, Kinesis.
• Mentor colleagues on best practices and technical concepts of building large scale solutions.

Benefits

• Medical insurance coverage
• Financial benefits including bonuses and long-term incentive units
• Flexible work arrangements
• Opportunities for professional development and training