About the role
Role Overview As a Remote Data Labeling Specialist, you will label and evaluate multi-modal AI training data (text, images, and conversations) to improve model performance in production systems. You will follow detailed rubrics, maintain strong annotation guidelines compliance, and contribute to measurable training data quality improvements.
What You Will Do
- Execute data labeling and data annotation for LLM training pipelines (classification, extraction, ranking)
- Perform RLHF workflows including preference comparisons, rationale tagging, and rubric-based scoring
- Conduct LLM evaluation and prompt evaluation for helpfulness, factuality, grounding, relevance, and instruction-following
- Apply named entity recognition and span labeling with strict rubric adherence
- Support computer vision annotation (bounding boxes, polygons, keypoints, image categorization) when needed
- Run QA evaluation audits, inter-annotator agreement checks, and targeted reviews to ensure training data quality
- Perform content safety labeling and policy-based moderation tagging
- Document edge cases, propose taxonomy updates, and escalate ambiguous items with evidence
Required Qualifications
- Experience with production data labeling or related QA evaluation workflows
- Comfort working with detailed taxonomies, rubrics, and policy language
- Ability to maintain throughput while protecting training data quality
- Clear asynchronous communication in a fully remote environment
Remote Work Details This is a full-time Remote role within the United States, with calibration sessions and QA checkpoints handled online.
You must follow confidentiality and secure dataset handling requirements.
Millions of jobs, with real people getting hired every day
Questions, answered
Click "Apply with JobAssist" – we tailor your resume and application to this role and submit it for your approval.
Yes. This role at Rex.zone was screened before publishing – we confirmed the employer before listing it.
The employer didn't disclose a salary range for this listing. JobAssist shows pay whenever it's available.
This position can be done from anywhere, with no in-office requirement.
Yes – every application is tailored from your profile and this job's requirements, and you can review and edit before it's sent.