Evals Specialist - Legal

This contractor will support the Frontier Evals team in executing a high-priority evaluation campaign by monitoring the end-to-end data collection process, performing quality control on incoming data, and helping maintain throughput without compromising standards. The role requires strong expertise in the legal field so the contractor can reliably judge whether tasks, deliverables, and rubrics reflect realistic workflows, feasible scopes, and appropriate expectations for high-quality work in that field.

Scope of responsibilities

Incoming Data Quality Control (QC)

• Review incoming tasks, deliverables, and rubrics to confirm they meet campaign standards for complexity, feasibility, completeness, clarity, and consistency.
• Review materials for domain realism, including correct terminology, realistic workflow structure, appropriate constraints, reasonable timelines, and deliverable expectations that match how work is actually performed in the field.
• Identify common failure modes such as vague or underspecified requests, unrealistic scope, missing context, unclear success criteria, inappropriate assumptions, or deliverables that do not match the underlying task.
• Provide actionable feedback to contributors and reviewers, and track rework or resubmission through closure.
• Help ensure that rubric criteria are specific, fair, and aligned to the actual task requirements rather than generic preferences.
• Flag cases where a task depends on inaccessible systems, privileged information, or non-portable internal knowledge unless such dependencies are explicitly intended and supported.

Data Collection Process Monitoring

• Monitor campaign health and throughput, including submission volume, rework rates, bottlenecks, turnaround time, and recurring quality issues.
• Proactively flag blocked contributors or reviewers, repeated sources of confusion, and drift from agreed standards or templates.
• Surface systematic issues to Frontier Evals leads and propose concrete mitigations, such as checklist updates, examples, clarifications, or changes to templates and guidance.
• Maintain lightweight QC logs, issue trackers, or audit trails to support retrospectives, analysis, and process improvement.

Contributor Support and Coordination (as needed)

• Answer contributor questions to improve first-pass quality and reduce avoidable back-and-forth.
• Coordinate with reviewers and subject matter experts on edge cases and escalate ambiguous decisions quickly.
• Provide weekly or ad hoc summaries of quality themes, common issue categories, open risks, and recommended updates to instructions or examples.
• Help calibrate contributors on what “good” looks like for tasks and rubrics in the target domain.

Technical Work (light to moderate)

• Use spreadsheets, light SQL, or simple Python scripts to spot anomalies, review metadata completeness, identify broken links or missing files, and support workflow optimization or automation.
• Assist with basic data hygiene and organization of uploaded materials, naming
• conventions, and record completeness.
• Additional tasks (as needed)
• Draft or revise campaign documentation, examples, QC checklists, and “good vs. bad” examples for the target domain.
• Participate in targeted QA sweeps or audits for high-risk slices, such as new task types, new contributors, or newly introduced rubric formats.
• Assist with tagging, categorization, and structured review of tasks or deliverables for downstream analysis.

Qualifications

• 3+ years of professional experience in the target domain or a closely related function.
• Strong judgment about what realistic, high-quality work looks like in that field.
• Excellent attention to detail and comfort applying standards and rubrics consistently.
• Able to write clear, direct feedback that improves quality quickly.
• Comfortable working in spreadsheets; familiarity with basic SQL or Python is helpful but not required.
• Able to work in ambiguous, fast-moving environments and make pragmatic judgment calls.

Pay Range: $80-$200/hour DOE

** 2 month contract to start