Sr. Engineering Manager, Data Acquisition (Foundations)

Openai - San Francisco, CA

Full-time - Senior

San Francisco, CA

Professional, Scientific, and Technical Services

About the position

The Senior Engineering Manager for Data Acquisition at OpenAI will lead the Data Acquisition team within the Foundations organization, responsible for all aspects of data collection to support training and scaling across the research organization. This role involves overseeing engineering initiatives, managing a team, and ensuring effective collaboration across various sub-teams to enhance data acquisition processes.

Responsibilities

• Lead and manage a team, driving engineering projects related to web crawling, data ingestion, and search capabilities.
• Oversee the architecture, development, and deployment of scalable distributed systems capable of handling petabytes of data.
• Collaborate closely with cross-functional teams, including Data Processing, Architecture, Scaling, and Legal, ensuring seamless integration, data flow, and system compliance with data privacy regulations.
• Provide technical leadership, mentoring, and career development to engineers within the team, fostering an inclusive and high-performing environment.
• Develop and maintain long-term strategic planning for data acquisition needs, identifying new opportunities for data collection and processing improvements.
• Partner with senior leadership to align team priorities with broader organizational goals and ensure successful execution of team initiatives.
• Ensure high reliability, scalability, and performance of data acquisition services, optimizing existing processes and introducing new technologies as appropriate.
• Advocate for best practices in software engineering, including robust testing, system monitoring, and continuous improvement of team processes.
• Oversee routine system checks and monitoring for backend services deployed in a Kubernetes Infrastructure-as-Code environment.

Requirements

• 10+ years of industry experience in software engineering, with a focus on large-scale distributed systems and data processing.
• 4+ years of experience in managing and leading engineering teams, with a proven ability to mentor and grow engineering talent.
• Strong expertise in building and scaling large web crawlers, data pipelines, and ingestion systems.
• Experience working with Kubernetes and Infrastructure-as-Code concepts.
• Demonstrated ability to drive complex, cross-team engineering projects and lead collaboration across different functions.
• Strong technical and strategic vision, with the ability to translate broad objectives into actionable engineering plans.
• Excellent communication skills, with the ability to effectively interact with technical and non-technical stakeholders.
• A passion for innovation and the ability to foster a culture of learning and continuous improvement.