Talent.com
Mindrift
AI Evaluation Engineer (Python, QA or Security)Mindrift • London, England, GB
No longer accepting applications
AI Evaluation Engineer (Python, QA or Security)

AI Evaluation Engineer (Python, QA or Security)

Mindrift • London, England, GB
30+ days ago
Salary
£50.00 hourly
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

AI Evaluation Engineer (Python, QA or Security) • London, England, GB

Similar jobs

AI / Machine Learning Engineer

Academy Education Network LtdGreater London, England, United Kingdom
Full-time

AI / ML Engineers build, evaluate, and deploy machine‑learning systems at scale.The day‑to‑day work combines data preprocessing, model training (PyTorch, TensorFlow), evaluation against business or... Show more

 • Promoted

Senior Applied ML Engineer (Agentic Search)

NebiusGreater London, England, United Kingdom
Full-time

Nebius is leading a new era in cloud infrastructure for the global AI economy.We are building a full‑stack AI cloud platform that supports developers and enterprises from data and model training th... Show more

 • Promoted

Remote AI/ML Engineer — LLMs & Generative AI

dionix.aiGreater London, England, United Kingdom
Remote
Full-time

DIONIX AI is seeking an AI/ML Engineer specializing in Large Language Models and Generative AI for its B2B SaaS products.You will design, build, and deploy agentic workflows, and predictive models,... Show more

 • Promoted

Staff Machine Learning Engineer (£220,000 | RecSys | Python)

DexGreater London, England, United Kingdom
Full-time

London, 4 days in office (Shoreditch).Up to £220,000 + Bonus + Compelling Equity.This role is with one of Dex’s trusted partner companies.We work closely with their teams to truly understand their ... Show more

 • Promoted

Lead AI Software Engineer - TRP Labs London

T. Rowe PriceGreater London, England, United Kingdom
Full-time

Lead AI Software Engineer - TRP Labs LondonApplylocations: London, Warwick Courttime type: Full timeposted on: Posted Todayjob requisition id: 82631At T.Rowe Price, we identify and actively... Show more

 • Promoted

Developer Experience Engineer for AI Hardware

FractileGreater London, England, GB
Full-time

Fractile is striving to redefine computing for AI, seeking Developer Experience Engineers who will design tools for running and optimising large language models.You will collaborate with various en... Show more

 • Promoted

Senior AI Engineer

Acrotrend - A NowVertical CompanyGreater London, England, United Kingdom
Full-time

Architect and lead the implementation of multi‑agent systems using Google AI SDKs (Vertex AI Agent Builder), LangGraph, CrewAI, and other emerging orchestration frameworks.Design and build stateful... Show more

 • Promoted

Senior MLOps Engineer: AWS AI Platform (On-site London)

Talenzon groupGreater London, England, United Kingdom
Temporary

Talenzon group is seeking a Senior MLOps Engineer to join our London-based on-site team for a 6-month contract.You will design and implement an enterprise MLOps platform on AWS, deploy production-r... Show more

 • Promoted

AI Software Engineer – Agent Orchestrator & Production AI

CloudMarginGreater London, England, United Kingdom
Full-time

CloudMargin is seeking a senior software engineer with deep AI interest to shape our AI-native product strategy.You will lead AI coding agents, orchestrate the engineering pipeline and deliver secu... Show more

 • Promoted

Generative AI Engineer

McCabe & BartonGreater London, England, United Kingdom
Permanent

London, hybrid, minimum 2 days per week in the office.McCabe & Barton is partnering with a leading insurance business to hire an Artificial Intelligence Engineer for a hands-on role focused on buil... Show more

 • Promoted • New!

RL Research Engineer: Safe, Scalable AI Systems

AnthropicGreater London, England, United Kingdom
Full-time

Anthropic is seeking a Research Engineer specializing in Reinforcement Learning to advance large language model capabilities.This role involves collaborative research and engineering, focused on op... Show more

 • Promoted

Senior AI/ML Engineer

Argus MediaGreater London, England, United Kingdom
Full-time

Great AI and machine learning capability only creates value once it runs reliably in the real world.This role exists to bridge that final, critical gap—taking the AI/ML solutions developed within o... Show more

 • Promoted

Senior AI Engineer

Chubb Ltd.Greater London, England, United Kingdom
Full-time

Chubb is looking for a Senior AI Engineer to lead the design, development, and deployment of advanced generative AI solutions that improve how we operate across the insurance lifecycle — from under... Show more

 • Promoted

Senior AI Engineer

CapcoGreater London, England, United Kingdom
Permanent

Senior AI Engineer (Principal Consultant).Technology & Engineering | Type: Permanent.Empower the next frontier of AI with Capco.Shape, build, and deliver future-ready generative systems.We’re looki... Show more

 • Promoted

VP Lead Software Engineer - AI-Driven XVA Systems

Citibank (Switzerland) AGGreater London, England, United Kingdom
Full-time

Citibank (Switzerland) AG is seeking a Lead Software Engineer (XVA) at the Vice President level in London.You will lead advanced software development and AI-driven innovation, working with high-cal... Show more

 • Promoted

Senior AI Engineer - Greenfield LLMs in Production

SaragossaGreater London, England, United Kingdom
Full-time

A cutting-edge AI company in the UK is seeking a senior professional to lead the development of innovative AI solutions.You will work on greenfield projects deploying LLMs into production, influenc... Show more

 • Promoted

AI Engineer

Solve IntelligenceGreater London, England, GB
Full-time

AI - we are building the proprietary algorithms that define the future of Intellectual Property.We have real paying users, massive traction, and a mission to build a billion-dollar company.We are l... Show more

 • Promoted

Senior AI Engineer

DV Trading LLCGreater London, England, United Kingdom
Full-time

DV Trading is building a centralized AI function to govern, standardize, and accelerate firm‑wide AI adoption.The Senior AI Engineer defines how the firm builds production agent systems and ships r... Show more

 • Promoted

Senior AI-Native Engineer: Tooling & Platform Enablement

Mimecast LimitedGreater London, England, United Kingdom
Full-time

Mimecast Limited in London is seeking a Senior Software Engineer to contribute to the build-and-prove layer of AI Native Engineering within the P&E group.You will deliver reusable tooling, agentic ... Show more

 • Promoted

DevOps Engineer - AI Model Evaluator

MercorLondon, England, United Kingdom
Remote
Part-time
Quick Apply

Headquartered in San Francisco, our investors include.DevOps / SRE / Cloud Engineer (Coding Agent Experience).Review model-generated implementations involving.Identify bugs, edge cases, reliability... Show more