Talent.com
Ai Agent Evaluation Analyst

Ai Agent Evaluation Analyst

MindriftJohannesburg, Gauteng, South Africa
17 hours ago
Job description

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What We Do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

Who we're looking for

We're looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil's advocate.

Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

This is a flexible, project-based opportunity well-suited for :

  • Analysts, researchers, or consultants with strong critical thinking skills
  • Students (senior undergrads / grad students) looking for an intellectually interesting gig
  • People open to a part-time and non-permanent opportunity

About the project

We're on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you'll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases.

You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you've ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit.

What you'll be doing

  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism
  • Identifying inconsistencies, missing assumptions, or unclear decision points
  • Helping define clear expected behaviors (gold standards) for AI agents
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives
  • Thinking through complex systems and policies as a human would to ensure agents are tested properly
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage
  • How to get started

    Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.

    Requirements

  • Excellent analytical thinking : Can reason about complex systems, scenarios, and logical implications
  • Strong attention to detail : Can spot contradictions, ambiguities, and vague requirements
  • Familiarity with structured data formats : Can read, not necessarily write JSON / YAML
  • Ability to assess scenarios holistically : What's missing, what's unrealistic, what might break?
  • Good communication and clear writing (in English) to document your findings.
  • We also value applicants who have :

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design
  • Background in consulting, academia, olympiads (e.g. logic / math / informatics), or research
  • Exposure to LLMs, prompt engineering, or AI-generated content
  • Familiarity with QA or test-case thinking (edge cases, failure modes, "what could go wrong")
  • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.)
  • Benefits

  • Get paid for your expertise, with rates that can go up to $20 / hour depending on your skills, experience, and project needs
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio
  • Influence how future AI models understand and communicate in your field of expertise
  • #J-18808-Ljbffr

    Create a job alert for this search

    Analyst • Johannesburg, Gauteng, South Africa

    Related jobs
    • Promoted
    Analyst Research Remote

    Analyst Research Remote

    Hustle Consulting (Pty) LtdJohannesburg, Gauteng, South Africa
    Remote
    The Strategic Partner Analyst will play a critical role in ensuring high Supply Partner satisfaction as they adopt and scale with our clients' Supply APIs and survey demand.This is an exciting oppo...Show moreLast updated: 30+ days ago
    Senior AI Research Engineer, Model Inference (100% Remote)

    Senior AI Research Engineer, Model Inference (100% Remote)

    Tether Operations LimitedJohannesburg, GP, ZA
    Remote
    Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution. Our cutting-edge solutions empower businesses—from ex...Show moreLast updated: 30+ days ago
    • Promoted
    AI Engineer

    AI Engineer

    The Vocation StationWorkFromHome, Gauteng, South Africa
    Cape Town | Full‑Time | Remote.Build AI Solutions That Deliver Real‑World Impact.This is a role where technical skill meets practical application – helping organizations harness AI to improve ope...Show moreLast updated: 30+ days ago
    • Promoted
    AI Agent Evaluation Analyst

    AI Agent Evaluation Analyst

    MindriftWorkFromHome, South Africa
    Get AI-powered advice on this job and more exclusive features.This opportunity is designed for candidates who are residing in the specified country, with eligibility and rates potentially affected ...Show moreLast updated: 23 days ago
    • Promoted
    AI Engineer (LLMs & Agents)

    AI Engineer (LLMs & Agents)

    iOLAP, Inc.WorkFromHome, South Africa
    Are you passionate about leveraging AI to transform global industries? Do you have a proven track record of working with LLMs and building agentic solutions? Join. We’re looking for proactive self-s...Show moreLast updated: 30+ days ago
    Evaluation Scenario Writer - AI Agent Testing Specialist

    Evaluation Scenario Writer - AI Agent Testing Specialist

    MindriftJohannesburg, Gauteng, ZA
    Remote
    Quick Apply
    This opportunity is only for candidates currently residing in the specified country.Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of En...Show moreLast updated: 4 days ago
    • Promoted
    AI Engineer (Technology / Mining)

    AI Engineer (Technology / Mining)

    Black Pen RecruitmentJohannesburg, South Africa
    Our client is a growing technology company that revolutionises mining logistics across South Africa.Their platform connects mines and hauliers, automating vehicle tracking, order management and wei...Show moreLast updated: 30+ days ago
    • Promoted
    Applications Engineer

    Applications Engineer

    Boardroom AppointmentsKempton Park, South Africa
    Work with suppliers to create detailed technical requirements, including system architecture, hardware, software, and data, ensuring risk management, quality considerations, and product deployment ...Show moreLast updated: 30+ days ago
    • Promoted
    Embedded Intelligence Analyst (SSA)

    Embedded Intelligence Analyst (SSA)

    Sibylline LtdJohannesburg, Gauteng, South Africa
    We are looking for an experienced intelligence analyst to join an embedded intelligence and security operations team for a professional services client based in Johannesburg South Africa.As the reg...Show moreLast updated: 19 days ago
    • Promoted
    Lead Data and AI Engineer

    Lead Data and AI Engineer

    Boardroom AppointmentsSandton, South Africa
    Drive the vision, execution, and continuous improvement of the company's data and AI team.Mentor a high-performing team, instilling best practices and innovative thinking.Align projects with busine...Show moreLast updated: 30+ days ago
    • Promoted
    Bee Verification Analyst Centurion

    Bee Verification Analyst Centurion

    People DimensionCenturion, Gauteng, South Africa
    A well-established verifications company based in Centurion is looking for a BEE Analyst / Verification Analyst to join their team. Proactively communicate and collaborate with clients to achieve set ...Show moreLast updated: 30+ days ago
    • Promoted
    Impact Evaluator

    Impact Evaluator

    Future of Life InstituteWorkFromHome, South Africa
    The Future of Life Institute (FLI) is hiring an.This role is crucial for systematically capturing grantee voices and providing data‑driven insights that will shape our strategy.The Impact Evaluator...Show moreLast updated: 7 days ago
    • Promoted
    AI Quality Analyst (Conversation Intelligence & Evaluation)

    AI Quality Analyst (Conversation Intelligence & Evaluation)

    DASH BPORandburg, Gauteng, South Africa
    AI Quality Analyst (Conversation Intelligence & Evaluation).We are seeking a detail-oriented and insight-driven AI Quality Analyst to evaluate the performance of AI-powered customer interactions ac...Show moreLast updated: 20 days ago
    • Promoted
    Accounting AI Analyst | South Africa | Remote

    Accounting AI Analyst | South Africa | Remote

    OperationsArmyJohannesburg, South Africa
    Staggered shifts between 8 : 00 AM to 8 : 00 PM EST.Were looking for highly detail-oriented.In this role, you will be responsible for labeling and reviewing AI-generated outputs, identifying inconsiste...Show moreLast updated: 30+ days ago
    • Promoted
    AI Engineer

    AI Engineer

    Air ChefsWorkFromHome, Gauteng, South Africa
    AI Operations & Strategy Engineer — Location : Remote (South Africa).Manage and refine prompt engineering strategies.Monitor and optimise AI resource usage and costs. Develop evaluation frameworks to...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Ai Research Engineer, Model Inference (100% Remote)

    Senior Ai Research Engineer, Model Inference (100% Remote)

    Tether Operations LimitedWorkFromHome, Gauteng, South Africa
    Remote
    Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution. Our cutting‑edge solutions empower businesses—from ex...Show moreLast updated: 8 days ago
    • Promoted
    Al Engineers (Gen AI) – Remote – up to R1.5mil per annum

    Al Engineers (Gen AI) – Remote – up to R1.5mil per annum

    E-MergeJohannesburg, South Africa
    Remote
    Join a company at the forefront of digital transformation leveraging on cutting-edge AI technologies to deliver innovative solutions across diverse industries. Become a key part of a dynamic, forwar...Show moreLast updated: 30+ days ago
    Evaluation Scenario Writer - QA

    Evaluation Scenario Writer - QA

    MindriftJohannesburg, Gauteng, ZA
    Remote
    Quick Apply
    We believe in using the power of collective human intelligence to ethically shape the future of AI.The Mindrift platform, launched and powered by. AI projects from innovative tech clients.Our missio...Show moreLast updated: 30+ days ago