Research Scientist Intern (Doubao (Seed) - Machine Learning System) - 2025 Summer (PhD)

$124,800 - $124,800/Yr

Bytedance - San Jose, CA

Full-time - Intern

San Jose, CA

Professional, Scientific, and Technical Services

About the position

Established in 2023, the ByteDance Doubao (Seed) Team is dedicated to building industry-leading AI foundation models. We aim to do world-leading research and foster both technological and social progress. With a long-term vision and a strong commitment to the AI field, the Team conducts research in a range of areas including natural language processing (NLP), computer vision (CV), and speech recognition and generation. It has labs and researcher roles in China, Singapore, and the US. Leveraging substantial data and computing resources and through continued investment in these domains, our team has built a proprietary general-purpose model with multimodal capabilities. In the Chinese market, Doubao models power over 50 ByteDance apps and business lines, including Doubao, Coze, and Dreamina, and was launched to external enterprise clients through Volcano Engine. The Doubao app is the most used AIGC app in China. The AML Machine Learning Systems team provides end-to-end machine learning experience and machine learning resources for the company. The team builds heterogeneous ML training and inference systems based on GPU and AI chips and advances the state-of-the-art of ML systems technology to accelerate models such as stable diffusion and large language models (LLM). The team is also responsible for research and development of hardware acceleration technologies for AI and cloud computing, via technologies such as distributed systems, compilers, high-performance computing (HPC), and RDMA networking. The team is reinventing the ML infrastructure for large scale language models. We have published papers at top tier conferences such as SIGCOMM, NSDI, EuroSys, OSDI, SOSP, MLSys, NeurIPS, etc. Internships at ByteDance aim to provide students with hands-on experience in developing fundamental skills and exploring potential career paths. A vibrant blend of social events and enriching development workshops will be available for you to explore. Here, you will utilize your knowledge in real-world scenarios while laying a strong foundation for personal and professional growth. This Internship Program runs for 12 weeks beginning in May/June 2025.

Responsibilities

• Research and develop our machine learning systems, including heterogeneous computing architecture, management, scheduling, and monitoring.
• Manage cross-layer optimization of system and AI algorithms and hardware for machine learning (GPU, ASIC).
• Implement both general purpose training framework features and model specific optimizations (e.g. LLM, diffusions).
• Improve efficiency and stability for extremely large scale distributed training jobs.

Requirements

• Currently in PhD program in distributed, parallel computing principles and know the recent advances in computing, storage, networking, and hardware technologies.
• Familiar with machine learning algorithms, platforms and frameworks such as PyTorch and Jax.
• Have basic understanding of how GPU and/or ASIC works.
• Expert in at least one or two programming languages in Linux environment: C/C++, CUDA, Python.
• Must obtain work authorization in country of employment at the time of hire, and maintain ongoing work authorization during employment.

Nice-to-haves

• GPU based high performance computing, RDMA high performance network (MPI, NCCL, ibverbs).
• Distributed training framework optimizations such as DeepSpeed, FSDP, Megatron, GSPMD.
• AI compiler stacks such as torch.fx, XLA and MLIR.
• Large scale data processing and parallel computing.
• Experiences in designing and operating large scale systems in cloud computing or machine learning.
• Experiences in in-depth CUDA programming and performance tuning (cutlass, triton).

Benefits

• Hourly rate of $60.
• 100% premium coverage for Full-Time intern medical insurance after 90 days from the date of hire.
• Paid holidays and paid sick leave.
• Mental and emotional health benefits through the Employee Assistance Program.
• Reimbursements for mobile phone expenses.