About the role
Job Description Job Description: Head of Evaluations
Location: New York, NY / Remote
About The Company Newcode.ai is transforming how law firms and legal professionals use AI. The team works at the intersection of AI and legal innovation, building products for the future of legal work.
Note: As a fast-growing startup, not every process or playbook is fully established. Priorities can shift quickly. The ideal candidate is comfortable with ambiguity, takes ownership, and proactively creates structure.
Compensation & Benefits Position Overview
Discover more
jobs
Educational Resources
Engineering & Technology
We are seeking a highly analytical professional with a strong statistical background to lead evaluations. You will design, implement, and scale testing frameworks to ensure AI products meet high standards for legal reasoning, factual accuracy, regulatory compliance, and near-zero hallucination rates.
Key Responsibilities
- Design Legal Benchmarks for Contract Drafting, Information Extraction, Legal Research, and Contract Review
- Build, source, and maintain relevant datasets
- Audit AI Output: Review and score complex AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision; lay out a strategy
- Define Evaluation Metrics: Establish clear criteria for grading model performance, focusing on logical reasoning, citation accuracy, and safe abstention
- Collaborate with Engineering: Translate legal errors into actionable technical feedback for model fine-tuning
Requirements
- PhD or Masters in statistics, mathematics, machine learning, or equivalent
- Analytical Skills: Proven ability to break down complex statutory frameworks and case law into structured, logical data points
- Tech-Savviness: Proficiency in Python, pandas, numpy, Jupyter notebooks, and similar statistical models
Millions of jobs, with real people getting hired every day
Questions, answered
Click "Apply with JobAssist" – we tailor your resume and application to this role and submit it for your approval.
Yes. This role at ChatGPT Jobs was screened before publishing – we confirmed the employer before listing it.
The employer didn't disclose a salary range for this listing. JobAssist shows pay whenever it's available.
This position can be done from anywhere, with no in-office requirement.
Yes – every application is tailored from your profile and this job's requirements, and you can review and edit before it's sent.