Newcode.ai Logo

Newcode.ai

Head of Evaluations (Legal AI Benchmarking)

Reposted 3 Months Ago
Remote
Hiring Remotely in United States
Senior level
Remote
Hiring Remotely in United States
Senior level
Lead design and scaling of legal AI evaluation frameworks and benchmarks, build and maintain datasets, audit and score AI legal outputs, define metrics focused on reasoning and citation accuracy, and collaborate with engineering to translate legal errors into model fine-tuning feedback.
The summary above was generated by AI
Who are we?

At Newcode.ai, we're transforming how law firms and legal professionals harness AI for real-world impact. As part of our collaborative, high-growth team, you'll have the rare opportunity to work side-by-side with visionary founders at the bleeding edge of AI and legal innovation — shaping not just our product, but the future of legal work itself.

Note: We believe in being transparent about what it's like to work at Newcode. As a fast-growing startup, we're building and evolving every day. That means not every process, playbook, or framework is already in place, and priorities can shift quickly.

The people who thrive here are comfortable with ambiguity, take ownership, and don't wait for perfect direction. They are resourceful, proactive, and able to "figure it out"—solving problems, creating structure where needed, and helping build the company as they go. If you are good with this then, great! Keep reading to learn more.

Position Overview 

We are seeking a highly analytical professional with a strong statistical background to join our Head of Evaluations. In this role, you will design, implement, and scale the testing frameworks used to evaluate our platform. You will ensure our AI products meet the highest standards of legal reasoning, factual accuracy, and regulatory compliance while maintaining a near-zero hallucination rate. 

Key Responsibilities 

  • Design Legal Benchmarks for: Contract Drafting, Information Extraction, Legal Research, and Contract Review 
  • Build, source and maintain relevant datasets 
  • Audit AI Output: Review and score complex AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision and lay out a strategy.  
  • Define Evaluation Metrics: Establish clear criteria for grading model performance, specifically focusing on logical reasoning, citation accuracy, and the model's ability to safely abstain from answering. 
  • Collaborate with Engineering: Partner directly with Engineering to translate legal errors into actionable technical feedback for model fine-tuning. 

Requirements
  • PhD or Masters in statistics, mathematics, machine learning or equivalent 
  • Analytical Skills: Proven ability to break down complex statutory frameworks and case law into structured, logical data points. 
  • Tech-Savviness: python, panda, numpy, jupiter notebooks and similar statistical models 

Visa Sponsorship:

At this time, Newcode is unable to provide visa sponsorship. Candidates must be authorized to work in the applicable country without employer sponsorship.

Similar Jobs

2 Hours Ago
Remote
USA
Junior
Junior
Artificial Intelligence • Consumer Web • Digital Media • Information Technology • Social Impact • Software
Supports day-to-day IT operations for a distributed team, including user access requests, onboarding and offboarding, SaaS inventory, documentation, troubleshooting, ticket management, procurement support, and compliance evidence collection. Uses established processes and runbooks, maintains accurate records, assists users with workplace applications and account issues, and escalates unfamiliar or sensitive requests while developing IT operations expertise.
Top Skills: 1PasswordGoogle SheetsGoogle WorkspaceIdentity And SsomacOSMfaExcelSaas Management SystemsSlackSoc 2TwingateWindowsZoom
3 Days Ago
In-Office or Remote
District of Columbia, USA
128K-167K Annually
Mid level
128K-167K Annually
Mid level
Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3 • Infrastructure as a Service (IaaS)
Build and own Finance data models covering ledgers, settlements, transactions, invoicing, fees, and collateral. Reconcile internal and external systems, investigate discrepancies, support month-end close and audits, automate reporting, implement data quality tests, define reusable financial metrics, and create self-service reporting. Partner with Finance, Accounting, data, and platform engineering teams to maintain trusted, documented data systems.
Top Skills: AirflowBigQueryDagsterDatastreamDbtFivetranLightdashLookerMeltanoOmniPythonRedshiftSnowflakeSQL
3 Days Ago
Remote or Hybrid
District of Columbia, USA
128K-167K Annually
Mid level
128K-167K Annually
Mid level
Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3 • Infrastructure as a Service (IaaS)
Build and maintain scalable batch and streaming data pipelines for payments, ledger, transaction, and blockchain data. Own ingestion from databases, files, APIs, and onchain sources; develop incremental dbt models; orchestrate workflows in Dagster; and implement data quality, freshness, reconciliation, monitoring, deduplication, and late-data handling. Partner with engineering, analytics, data science, and risk teams to support reliable reporting and machine-learning features.
Top Skills: AirbyteAirflowApache BeamBigQueryCdcClickhouseDagsterDataflowDatastreamDbtDebeziumEvmFivetranFlinkGCPGcsKafkaMeltanoPub/SubPythonRustSftpSnowflakeSolanaSQL

What you need to know about the Chicago Tech Scene

With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.

Key Facts About Chicago Tech

  • Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
  • Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
  • Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
  • Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account