SEAL Research Scientist/ Research Engineer

Scale Ai - San Francisco, CA

Full-time - Mid Level

San Francisco, CA

Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services

About the position

The SEAL Research Scientist/Research Engineer at Scale AI, Inc. is a pivotal role focused on enhancing the safety and transparency of AI technologies. This position involves designing and implementing robust evaluation benchmarks, conducting cutting-edge research in model-assisted red teaming, and developing automated rating systems using large language models (LLMs). The role is integral to shaping the future of AI evaluation and ensuring trustworthy AI deployments across the industry.

Responsibilities

• Design and implement robust evaluation benchmarks.
• Measure and improve evaluation reproducibility and reliability.
• Research and implement cutting-edge model-assisted red teaming methods.
• Develop state-of-the-art LLM-based automated rating systems leveraging the Scale ecosystem for customized training data.
• Develop rater assist techniques such as critique modeling for improving rating quality.

Requirements

• Commitment to promoting safe, secure, and trustworthy AI deployments.
• Practical experience conducting technical research collaboratively, with proficiency in frameworks like Pytorch, Jax, or Tensorflow.
• A track record of published research in machine learning, particularly in generative AI.
• At least three years of experience addressing sophisticated ML problems in a research setting or product development.
• Strong written and verbal communication skills to operate in a cross-functional team.

Nice-to-haves

• Hands-on experience with open source LLM fine-tuning or involvement in bespoke LLM fine-tuning projects using Pytorch/Jax.
• Experience in crafting evaluations or a background in data science roles related to LLM technologies.
• Experience working with cloud technology stack (e.g., AWS or GCP) and developing machine learning models in a cloud environment.

Benefits

• Comprehensive health, dental, and vision coverage.
• Retirement benefits.
• Learning and development stipend.
• Generous PTO.
• Potential commuter stipend.