Senior Site Reliability Engineer, ML System - Foundation Model

About the position

ByteDance, founded in 2012, is on a mission to inspire creativity and enrich life through its diverse suite of products, including TikTok and various platforms tailored for the Chinese market. The company emphasizes the importance of creation, innovation, and teamwork in achieving its goals. The Doubao (Seed) Team, established in 2023, focuses on pioneering advanced AI foundation models, with research areas spanning deep learning, reinforcement learning, and AI safety. The Machine Learning (ML) System sub-team is dedicated to developing and maintaining distributed ML training and inference systems globally, ensuring high performance and reliability.

Responsibilities

• Ensure ML systems operate efficiently for large model development, training, evaluation, and inference.
• Maintain stability of offline tasks/services across multi-data center, multi-region, and multi-cloud scenarios.
• Manage resources and planning, including computing and storage resources, while overseeing costs and budgets.
• Oversee global system disaster recovery, cluster machine governance, and improve resource utilization and operational efficiency.
• Build software tools, products, and systems to monitor and manage ML infrastructure and services effectively.
• Participate in global team on-call support for system and business.

Requirements

• Strong experience in system engineering and machine learning.
• Proficiency in managing distributed systems and large-scale ML training environments.
• Experience with resource management and planning in cloud environments.
• Knowledge of disaster recovery and business service stability practices.
• Ability to build monitoring and management tools for ML infrastructure.

Nice-to-haves

• Experience with GPU/NPU/RDMA/Storage integration.
• Familiarity with AIGC/AGI systems and their operational requirements.
• Previous experience in a global team environment.

Benefits

• Opportunity to work with cutting-edge AI technologies.
• Collaborative work environment with a global team.
• Access to substantial data and computing resources for research.