Evaluation Scenario Writer - QA

Overview

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

Who we're looking for : We're looking for curious and intellectually proactive contributors who never miss an error and can think outside of the box when brainstorming solutions. Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

About the project — We’re on the hunt for an Evaluation Scenario Writer - QA for a new project focused on ensuring the quality and correctness of evaluation scenarios created for LLM agents. This project blends manual scenario validation, automated test thinking, and collaboration with writers and engineers. You will verify test logic, flag inconsistencies, and help maintain a high bar for evaluation coverage and clarity.

What you'll be doing

• Reviewing and validating test scenarios from Evaluation Writers
• Spotting logical inconsistencies, ambiguities, or missing checks
• Suggesting improvements to structure, edge cases, or scoring logic
• Collaborating with infrastructure and tool developers to automate parts of the review
• Creating clean and testable examples for others to follow

Although we're only looking for experts for this current project, contributors with consistent high-quality submissions may receive an invitation for ongoing collaboration across future projects.

Requirements

The ideal contributor will have :

We also value applicants who have :

Benefits

#J-18808-Ljbffr