Tesla - Palo Alto, CA

posted 5 months ago

Full-time - Mid Level
Palo Alto, CA
Transportation Equipment Manufacturing

About the position

Tesla's Supercomputing/AI infrastructure team is at the forefront of high-performance computing and machine learning infrastructure, which is essential for the operation of our machine learning algorithms. This encompasses a wide range of applications, including virtual simulations, Autopilot hardware, silicon design, and the Dojo supercomputer. As the demand for data and optimized compute resources continues to grow, our cluster builds are becoming larger and more complex. Therefore, the ongoing development and automation of deployment, monitoring, self-healing, and alerting processes are critical to the success of our engineering teams. The importance of this team and its contributions will only increase as we scale our Full-Self-Driving (FSD) and Robotaxi initiatives. In the role of Site Reliability Engineer, you will play a vital part in maintaining and enhancing our infrastructure to ensure that engineering teams across Autopilot, AI, and Dojo have the necessary tools and resources to maximize their productivity. Your responsibilities will include managing and operating our high-performance computing (HPC) clusters, monitoring compute, GPU, and network metrics, troubleshooting Linux systems, and performance tuning. You will also collaborate closely with our Data Center team to ensure the smooth operation of hundreds of servers and the successful deployment of new GPU capacity. Your efforts will directly support neural network training at scale, streamline the development of FSD, and help Dojo achieve its goal of becoming the most powerful supercomputer to date.

Responsibilities

  • Support the AI/ML cluster infrastructure on both GPU and Dojo platforms, focusing on systems automation, configuration management and deployment at scale
  • Improve our cluster health monitoring and auto-recovery pipeline
  • Work with users on debugging application performance issues
  • Collaborate with hardware and storage vendors to tune and optimize our servers, storage and network
  • Write Ansible playbooks for configuration management
  • Perform performance tuning and OS provisioning on Linux systems
  • Manage HPC clusters, workloads and applications
  • Engage in automation and systems engineering using Python, Golang or Bash/Shell
  • Participate in 24x7 on-call rotation

Requirements

  • Proficiency in Python, Golang and/or Bash
  • Proficiency with Linux fundamentals and performance optimizations (Ubuntu/RHEL OS)
  • Demonstrable knowledge of TCP/IP, IPoIB, Linux operating system internals, filesystems, disk/storage technologies and storage protocols
  • Experience collaborating with network and data center teams for large scale cluster builds
  • Experience with configuration management software (Ansible, etc.), systems monitoring & alerting (Prometheus, Grafana, Telegraf, Splunk, etc.), and/or administering HPC workload managers
  • Experience with high-throughput low-latency networks, GPU-based computing systems, and/or high performance storage systems
  • Experience with Slurm, LSF and storage management of distributed parallel file systems is a plus
  • Bachelor's Degree in Computer Science, Computer Engineering, Electrical Engineering, Physics or proof of exceptional skills in related field
  • 3+ years of additional equivalent experience or evidence of exceptional ability related to the position

Nice-to-haves

  • Experience with Slurm, LSF and storage management of distributed parallel file systems

Benefits

  • Aetna PPO and HSA plans with 2 medical plan options and $0 payroll deduction
  • Family-building, fertility, adoption and surrogacy benefits
  • Dental (including orthodontic coverage) and vision plans with $0 paycheck contribution options
  • Company Paid HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA
  • Healthcare and Dependent Care Flexible Spending Accounts (FSA)
  • LGBTQ+ care concierge services
  • 401(k) with employer match
  • Employee Stock Purchase Plans
  • Company paid Basic Life, AD&D, short-term and long-term disability insurance
  • Employee Assistance Program
  • Sick and Vacation time (Flex time for salary positions), and Paid Holidays
  • Back-up childcare and parenting support resources
  • Voluntary benefits including critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
  • Weight Loss and Tobacco Cessation Programs
  • Tesla Babies program
  • Commuter benefits
  • Employee discounts and perks program
© 2024 Teal Labs, Inc
Privacy PolicyTerms of Service