LLM Data Quality Analyst

US Tech Solutions - Sunnyvale, CA

Full-time

Sunnyvale, CA

Administrative and Support Services

About the position

The LLM Data Quality Analyst will play a crucial role in ensuring the quality of data produced for Large Language Models (LLMs). This position involves close collaboration with technical modeling teams and vendor leads to define data requirements, design instructions for content creators, and analyze datasets to maintain high quality standards. The analyst will utilize both quantitative and qualitative techniques to assess and improve data quality, while also communicating findings and best practices to stakeholders.

Responsibilities

• Define data requirements based on a deep understanding of the modeling team goals and study of model loss patterns.
• Design detailed instructions for content creator teams, illustrating nuanced differences with clear examples.
• Rapidly iterate with content-production vendors on small batches of data to arrive at desired data quality.
• Drive processes to deliver benchmark datasets for assessing data quality.
• Benchmark vendor data quality against competitor models.
• Use quantitative techniques to analyze vendor-produced data to assure high quality and drive rater pool optimization.
• Audit datasets for quality issues and develop tools for accelerating qualitative analysis.
• Use modeling and experimentation techniques to demonstrate data impact and identify blind spots.
• Send out regular reports to project stakeholders on data quality.
• Track key metrics such as iteration time and vendor quality audit results.
• Author best practices and guideline documents.

Requirements

• 3+ years of experience in Technical Writing, Quality Assurance or a related field.
• Excellent writing and editing skills, with the ability to adapt to different styles and tones.
• Strong analytical and problem-solving skills.
• Ability to work independently and as part of a team.

Nice-to-haves

• Experience with LLM processes like Pre-training, RLHF, SFT, Evals.