About the role
We are looking for a detail-oriented AI QA Engineer to join our AI Metadata Augmentation team. If you are passionate about ensuring the quality, accuracy, and reliability of AI/LLM outputs, this is a great opportunity to work on cutting-edge AI systems deployed on AWS.
Key Responsibilities
- Design and execute LLM evaluation & benchmarking strategies (relevancy, consistency, metadata quality)
- Build and maintain automated testing frameworks using Python
- Perform prompt regression testing to track impact of changes over time
- Collaborate with AI engineers to debug edge cases and improve prompts/logic
Required Skills & Experience
- Strong experience in QA Automation / Data Quality / Software Testing
- Understanding of AI lifecycle (Training vs Inference, Prompt Engineering)
- Proficiency in Python for test automation and data validation
- Hands-on experience with JSON parsing and validation
- Basic knowledge of AWS (CloudWatch, S3)
Millions of jobs, with real people getting hired every day
Questions, answered
Click "Apply with JobAssist" – we tailor your resume and application to this role and submit it for your approval.
Yes. This role at Tekskills was screened before publishing – we confirmed the employer before listing it.
The employer didn't disclose a salary range for this listing. JobAssist shows pay whenever it's available.
This position can be done from anywhere, with no in-office requirement.
Yes – every application is tailored from your profile and this job's requirements, and you can review and edit before it's sent.