Talent.com
Mindrift
AI Evaluation Engineer (Python, QA or Security)Mindrift • Birmingham, England, GB
AI Evaluation Engineer (Python, QA or Security)

AI Evaluation Engineer (Python, QA or Security)

Mindrift • Birmingham, England, GB
16 days ago
Salary
£50.00 hourly
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for

  • 5+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation

Up to $50/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Create a job alert for this search

AI Evaluation Engineer (Python, QA or Security) • Birmingham, England, GB

Similar jobs

Senior Software Engineer - Conversational AI in Healthcare

Transform9birmingham, england, United Kingdom
Full-time

A healthcare technology company in Birmingham is looking for a Senior Software Engineer to enhance their innovative conversational agent platform.The role involves designing and implementing scalab... Show more

 • Promoted

Design Evaluation Engineer

ChiprightTipton, England, GB
Full-time

Design Evaluation Engineer: Key Responsibilities:.Perform detailed design evaluations and testing of electronic hardware.Run, debug, and create VIs (Virtual Instruments) using LabView for bench tes... Show more

 • Promoted

Research Engineer - Power Electronics & AI

James Adamsbirmingham, england, United Kingdom
Full-time

A global engineering and technology organisation at the forefront of electrification and advanced transportation systems is seeking a.Power Electronics & AI Engineer.This is a unique opportunity to... Show more

 • Promoted

Software Engineer – Security & Intelligence

Envitia Groupcheltenham, england, United Kingdom
Full-time

Join the Growth Story at Envitia.Envitia operates at the heart of the UK’s most sensitive and high-profile operations.Our approach is entirely mission-focused, enabling customers to see more, decid... Show more

 • Promoted

AI Engineer

ITonlinelearning Recruitmentwolverhampton, (region), United Kingdom
Full-time

AI Trainee Engineer – £45,000–£75,000 – Job Programme.This is a self-funded programme that leads to employment, fees apply.Complete the programme and get a job, or get your course fees back.Looking... Show more

 • Promoted • New!

Forward Deployed AI Engineer

ACCENTUREBirmingham, England, GB
Full-time

It is not a project delivery role.A Forward Deployed AI Engineer is a production engineer who works embedded inside a client's enterprise, shoulder to shoulder with their teams, to make complex AI ... Show more

 • Promoted

Senior Full-Stack AI Engineer: Build Production-Ready AI

Amberesbirmingham, england, United Kingdom
Full-time

A leading technology company is seeking a Full Stack Engineer to join their team in Birmingham.The ideal candidate will have 5+ years of experience in software engineering and be proficient in Reac... Show more

 • Promoted

AI Platform Engineer: Agentic LLMs & Production

Goldman Sachs Group, Inc.birmingham, england, United Kingdom
Full-time

Asset & Wealth Management Software Engineer in Birmingham to lead AI projects that enhance program delivery.The role demands a strong background in software development, particularly in Python and ... Show more

 • Promoted

Machine Learning Implementation Engineer, AI (SFIA4)

Zaizi Ltdcheltenham, england, United Kingdom
Permanent

DV Clearance: Required & British.Work on exciting public sector projects and make a positive difference in people’s.At Zaizi, we thrive on solving complex challenges through creative thinking.As a ... Show more

 • Promoted

Senior AI Solutions Engineer - Python, FastAPI, Azure

Gateley PlcBirmingham, England, GB
Full-time

A leading legal and professional services group in Birmingham is seeking an experienced AI Developer to design and deploy AI-powered applications.The role involves building scalable AI services usi... Show more

 • Promoted

AI SAAS DEVELOPER

Felix Consultantsbirmingham, england, United Kingdom
Full-time

We are looking for an innovative AI Developer to design, develop, and deploy AI-powered applications using Large Language Models (LLMs), Natural Language Processing (NLP), and modern AI frameworks.... Show more

 • Promoted

Applied AI Engineer — ML Models & Production

Spot2itBirmingham, England, GB
Full-time

We are seeking a skilled AI Engineer to design and deploy machine learning models and intelligent solutions for our clients.Designed and developed AI/ML algorithms.Built and optimized predictive mo... Show more

 • Promoted

Software AI Engineer

SF Technology Partnersbirmingham, england, United Kingdom
Full-time

Software AI Engineer with generative AI, LLM or ML experience is sought by a high growth scale up based near Birmingham.Working at the forefront of national security innovation this Software AI Eng... Show more

 • Promoted

Hiring for AI Engineer

ThoughtWin - A CMMI Level 3 CompanyTipton, England, GB
Full-time

WE ARE HIRING! Were expanding our team and looking for talented individuals to join us at techugo- AI Engineers to build the next generation of AI-powered mobile applications.Days working and fixed... Show more

 • Promoted

Engineering AI Solutions Engineer

DoistBirmingham, England, GB
Full-time

Neural Concept Engineering seeks a Solution Engineer to bridge engineering credibility and strategic influence.You will drive pre-sales from discovery to SOW, serving as the technical compass throu... Show more

 • Promoted

Security‑Focused Data Engineer for Intelligence Analytics

Envitiacheltenham, england, United Kingdom
Full-time

Envitia is seeking a Data Engineer to join their Security and Intelligence team in Cheltenham.The successful candidate will design and maintain secure, scalable data platforms and pipelines, facili... Show more

 • Promoted

AI Engineer – LLMs, Agents & Applied Data Science

Talanbirmingham, england, United Kingdom
Full-time

Talan is seeking an AI Engineer to work on client-focused solutions in Birmingham.The role involves building machine learning components, mentoring juniors, and collaborating with senior staff.With... Show more

 • Promoted

AI Engineer

SCCbirmingham, england, United Kingdom
Permanent

We are actively building diverse teams and welcome applications from everyone.Birmingham (SCC operates hybrid working, which comprises a mix of office and home working).Excellent package: solid bas... Show more

 • Promoted

Machine Learning Implementation Engineer, AI (SFIA4)

Zaizicheltenham, england, United Kingdom
Permanent

DV Clearance: Required & British.Work on exciting public sector projects and make a positive difference in people’s.At Zaizi, we thrive on solving complex challenges through creative thinking.As a ... Show more

 • Promoted

Senior Global Threat Intelligence Platform Engineer

PwC UKBirmingham, England, GB
Full-time

A global consulting firm is seeking a developer to join their Threat Intelligence practice in Birmingham.The role involves building automation tools and APIs to support intelligence analysis.Candid... Show more