Create clear, structured rater guidelines, evaluation rubrics, and reviewer instructions for human evaluators across domains. Review and revise documentation to remove ambiguity, design scoring frameworks, handle edge cases, and collaborate with researchers and SMEs to ensure consistent, high-quality AI evaluations.
This role is for one of our clients
Compensation: $45-$65 per hour
Join a cutting-edge AI initiative focused on building the next generation of foundational AI models. We are seeking experienced Linguists, Instructional Designers, Technical Writers, and AI Content Specialists who excel at transforming complex, ambiguous program requirements into clear, structured, and actionable guidance for human evaluators.
In this role, you will play a critical part in improving AI training quality by creating precise rating guidelines, evaluation rubrics, and reviewer instructions across diverse domains including finance, retail, insurance, legal, and sports. Your expertise will help ensure consistency, accuracy, and reliability in human evaluations that directly shape advanced AI systems.
This is a full-time, fully remote engagement requiring a commitment of approximately 35 hours per week during standard weekdays.
RequirementsKey Responsibilities
- Translate complex and ambiguous program requirements into clear, concise, and easy-to-follow rater guidelines.
- Design comprehensive evaluation rubrics and scoring frameworks that enable consistent decision-making across a variety of AI tasks.
- Review existing documentation to identify ambiguity, inconsistencies, contradictions, and coverage gaps, then revise guidelines to improve clarity and usability.
- Convert subject-matter requirements from domains such as finance, retail, insurance, legal, and sports into structured evaluation instructions that non-domain raters can apply confidently.
- Collaborate with research teams, program managers, and subject matter experts to maintain consistency across multiple guideline sets.
- Develop documentation that addresses edge cases, exceptions, and complex evaluation scenarios while minimizing reviewer escalation.
- Minimum 3 years of professional experience in Linguistics, Instructional Design, Technical Writing, AI Content Development, or a closely related field.
- Direct experience creating, refining, or maintaining evaluation guidelines, rating rubrics, or reviewer instructions within Generative AI, RLHF, human evaluation, or AI data annotation environments.
- Demonstrated ability to work across multiple subject areas and convert domain-specific knowledge into clear, structured documentation.
- Proven experience resolving ambiguity and improving written specifications with measurable before-and-after improvements.
- Strong analytical thinking with exceptional attention to detail.
- Demonstrated career progression and professional growth.
- Ability to commit reliably to 35+ hours per week during weekdays.
- Outstanding written communication skills with the ability to explain nuanced concepts in a precise, consistent, and easy-to-understand manner.
- Experience supporting AI model training, evaluation, RLHF, or large-scale annotation programs.
- Background working with cross-functional teams including researchers, engineers, product managers, and subject matter experts.
- Familiarity with structured documentation standards, quality assurance methodologies, and guideline governance.
- Experience designing documentation that supports scalable, high-quality human evaluation processes.
- Help define how human evaluators assess next-generation AI systems.
- Influence the quality and consistency of AI training data across multiple industries.
- Collaborate with multidisciplinary experts working on advanced Generative AI initiatives.
- Solve complex language and reasoning challenges while improving AI evaluation standards at scale.
- Enjoy the flexibility of a fully remote engagement while contributing to high-impact AI research.
We are committed to creating an inclusive workplace and welcome applications from qualified professionals of all backgrounds. Reasonable accommodations are available throughout the application and engagement process.
Contract & Engagement Details- Independent contractor engagement.
- Fully remote with flexible working arrangements.
- Expected commitment of approximately 35 hours per week during weekdays.
- Project duration may vary depending on business requirements and individual performance.
- Work does not require access to confidential or proprietary information from any current or former employer.
- Payments are issued weekly based on approved work completed.
- At this time, we are unable to support H1-B or STEM OPT candidates.
Similar Jobs
Fintech • Financial Services
Own client implementation projects from contract signing through go-live. Build and maintain project plans, track risks and blockers, coordinate internal and client stakeholders, manage communication and expectations, secure required documents and approvals, escalate delays, and improve implementation processes. Serve as the client’s primary point of contact while coordinating technical and account teams without owning technical decisions or commercial relationships.
Top Skills:
APIsB2B Saas
Fintech • Financial Services
Execute Bloom Credit’s B2B marketing engine across ABM, content, social media, events, sales enablement, and digital channels. Manage HubSpot workflows and data hygiene, maintain marketing assets and partner toolkits, coordinate trade shows, webinars, newsletters, and collateral, track competitor activity, reconcile marketing expenses, and report weekly on website, traffic, conversion, and campaign metrics.
Top Skills:
BomboraCanvaFi NavigatorFigmaGoogle AnalyticsHubspot CrmLinkedInLinkedin Sales NavigatorPowerPointPpcSeo
Fintech • Financial Services
Own day-to-day IT and information security operations for a fully remote Mac fleet. Responsibilities include endpoint lifecycle management, identity and access administration, Google Workspace, helpdesk support, asset coordination, CrowdStrike EDR monitoring, DLP controls, Datadog SIEM alerting, security hygiene, access reviews, and SOC 2 and PCI evidence collection. The role requires autonomous ownership, automation, clear documentation, cross-functional collaboration, and approximately 4–6 years of experience beyond ticket-only support.
Top Skills:
CrowdstrikeDatadogDlpGoogle WorkspaceJumpcloudmacOSMdmPci DssSIEMSoc 2SsoZero Trust
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
