Talent.com
This job offer is not available in your country.
AI Agent Evaluation Analyst - AI Trainer

AI Agent Evaluation Analyst - AI Trainer

MindriftWorkFromHome, KwaZulu-Natal, South Africa
6 days ago
Job description

Overview

AI Agent Evaluation Analyst - AI Trainer role at Mindrift. We are looking for curious, intellectually proactive contributors who double-check assumptions and think through scenarios and edge cases. This is a flexible, project-based opportunity for those who enjoy evaluating how modern AI systems are tested and evaluated.

Mindrift connects domain experts with AI projects, powered by Toloka, to unlock the potential of GenAI through real-world expertise from across the globe.

What you'll do

  • Review evaluation tasks and scenarios for logic, completeness, and realism
  • Identify inconsistencies, missing assumptions, or unclear decision points
  • Help define clear expected behaviors (gold standards) for AI agents
  • Annotate cause–effect relationships, reasoning paths, and plausible alternatives
  • Think through complex systems and policies to ensure agents are tested properly
  • Work closely with QA, writers, or developers to suggest refinements or edge case coverage

Requirements

  • Excellent analytical thinking : ability to reason about complex systems, scenarios, and implications
  • Strong attention to detail : can spot contradictions, ambiguities, and vague requirements
  • Familiarity with structured data formats : can read JSON / YAML (not necessarily write)
  • Capability to assess scenarios holistically : identify what's missing or unrealistic and what might break
  • Good communication and clear writing in English to document findings

    • We also value

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design
  • Background in consulting, academia, Olympiads (logic / math / informatics), or research
  • Exposure to LLMs, prompt engineering, or AI-generated content
  • Familiarity with QA or test-case thinking (edge cases, failure modes)
  • Understanding of scoring / evaluation in agent testing (precision, coverage)

    • Benefits

  • Get paid for your expertise, with rates that can go up to $20 / hour depending on skills and project needs
  • Flexible, remote, freelance project that fits around your commitments
  • Gain valuable experience on an advanced AI project to enhance your portfolio
  • Influence how future AI models understand and communicate in your field

    • How to get started

    Apply to this post, qualify, and contribute to a project aligned with your skills on your own schedule. Shape the future of AI while building tools that benefit everyone.

    Seniorities

  • Internship

    • Employment type

  • Part-time

    • Job function

  • Other

    • Industries

  • IT Services and IT Consulting

    • Referrals increase your chances of interviewing at Mindrift.

    #J-18808-Ljbffr

    Create a job alert for this search

    Ai • WorkFromHome, KwaZulu-Natal, South Africa