Generative AI Automation Tester - Remote / Telecommute

Cynet Systems - Dallas, TX

Full-time - Entry Level

Remote - Dallas, TX

Professional, Scientific, and Technical Services

About the position

The Generative AI Automation Tester will play a crucial role in evaluating and testing Generative AI proof of concept (POC) models that are built using Open AI and Vertex LLM Models. This position is fully remote, allowing for flexibility while working on cutting-edge AI technologies. The tester will be responsible for designing, developing, and executing detailed test plans that validate the performance of these models using real customer data. This involves a thorough understanding of the data flow through AI models, ensuring that the input and output data are accurate and complete. In this role, collaboration is key. The tester will work closely with data scientists and engineers to ensure that the outputs generated by the models align with the expected results. This includes creating comprehensive test cases that assess various aspects of the models, such as accuracy, bias, performance, and edge cases. The tester will also be tasked with identifying weaknesses and inaccuracies within the models, as well as areas that require optimization. Reporting bugs, issues, and potential improvements will be a significant part of the job, requiring detailed feedback to be provided to the development teams. Additionally, the tester will utilize automated testing tools and frameworks specifically designed for model testing. Maintaining comprehensive documentation of the quality assurance (QA) process and results is essential to ensure transparency and facilitate future testing efforts.

Responsibilities

• Evaluate and test Generative AI POC models built using Open AI and Vertex LLM Models.
• Design, develop, and execute detailed test plans to validate the model performance with real customer data.
• Perform data validation, ensuring accuracy and completeness of the input/output data flow through AI models.
• Collaborate with data scientists and engineers to ensure the model's output is aligned with the expected results.
• Create test cases to assess model accuracy, bias, performance, and edge cases.
• Identify model weaknesses, inaccuracies, and areas for optimization.
• Report bugs, issues, and improvement areas, providing detailed feedback to development teams.
• Use automated testing tools and frameworks for model testing.
• Maintain comprehensive documentation of the QA process and results.

Requirements

• Proven experience in testing Machine Learning/AI models.
• Familiarity with Generative AI models like Open AI GPT, Vertex AI models.
• Strong knowledge of data validation, model performance, and quality assurance practices.
• Experience with model evaluation metrics such as accuracy, precision, recall, F1 score, and bias analysis.
• Proficiency in Python or other relevant programming languages.
• Familiarity with testing frameworks and automated testing tools for AI models.
• Strong analytical and problem-solving skills.

Nice-to-haves

• Experience in testing AI models with real-world datasets.
• Knowledge of model versioning, deployment, and monitoring.