Senior Data Engineer

Disney - Santa Monica, CA

Full-time - Mid Level

Santa Monica, CA

Motion Picture and Sound Recording Industries

About the position

At Disney Entertainment & ESPN Technology, we are dedicated to reimagining the way audiences experience our beloved stories through innovative technology. Our Search Machine Learning (ML) team plays a crucial role in enhancing the search capabilities of Disney+ and Hulu platforms. This position involves working in a highly collaborative environment where ML-based services are integrated throughout the entire search journey, from understanding user queries to semantic retrieval and engagement-based content re-ranking. We are committed to constant experimentation, testing new ideas, and implementing successful strategies to improve user experience. In this role, you will partner closely with ML engineers and data scientists to automate and manage their data needs for regular training and online inference. You will take ownership of the entire lifecycle of data pipelines, ensuring efficient communication with upstream data teams. Your expertise in data engineering will be vital in bringing innovative ideas to life by designing and developing effective data pipelines that support the feature requirements of our ML engineers and data scientists. The Commerce, Growth & Identity (CGI) team, which you will be a part of, focuses on creating seamless user experiences and building innovative capabilities that drive subscriptions and engagement across Disney's streaming and digital products. This role requires collaboration with engineers from diverse backgrounds and skill sets, working in an Agile environment that emphasizes teamwork and collaboration.

Responsibilities

• Design and develop offline (batch) and near-line (streaming) data jobs.
• Collaborate with ML and data practitioners to automate their pipelines.
• Build tooling and low-latency services to enable and support event-driven architectures.
• Work on multi-faceted projects with engineers from diverse backgrounds and skills across teams.
• Participate in an Agile environment that focuses on collaboration and teamwork.

Requirements

• 5+ years of software experience, with 3+ years of relevant data and software experience.
• Knowledge of the Python/Java data ecosystem.
• Experience in building large datasets and scalable services.
• Experience deploying and running services in AWS, and engineering big-data solutions using technologies like Databricks, EMR, S3, Spark.
• Experience loading and querying cloud-hosted databases such as Redshift and Snowflake.
• Experience designing and developing backend microservices for large scale distributed systems using gRPC or REST.
• Experience with large-scale distributed data processing systems and cloud infrastructure such as AWS or GCP.

Nice-to-haves

• Knowledge in optimizing Spark jobs.
• Experience building streaming pipelines using Kafka, Spark, Kinesis.
• Mentor colleagues on best practices and technical concepts of building large scale solutions.

Benefits

• Medical insurance coverage
• Financial benefits including bonuses and long-term incentive units
• Flexible work arrangements
• Opportunities for professional development and training