Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • Manchester, England, GB
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • Manchester, England, GB
25 days ago
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Create a job alert for this search

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • Manchester, England, GB

Similar jobs

Senior Python Backend Engineer - Predictive AI Platform

Anaplan IncManchester, England, GB
Full-time

Anaplan Inc in Manchester is seeking a Senior Backend Engineer to join its Predictive Intelligence engineering team.You will design and build backend services that support predictive and forecastin... Show more

 • Promoted

Remote AI Engineer: ML, AWS & Kubernetes

YO IT ConsultingManchester, England, GB
Remote
Full-time

YO IT Consulting is seeking an AI Engineer for a remote contract role.You will train next-generation AI systems and design scalable ML pipelines using AWS services and Kubernetes.Strong programming... Show more

 • Promoted

Senior Python AI Architect: Autonomous Agents

ZippdManchester, England, GB
Full-time

Zippd is seeking a Senior Python Developer in Manchester to lead and architect groundbreaking AI initiatives.You will oversee the development of autonomous agents aimed at solving complex logistica... Show more

 • Promoted

Senior Python Engineer — Lead, Mentor & Deliver

Hippo DigitalManchester, England, GB
Full-time

Hippo Digital, based in Manchester, is looking for a Senior Python Developer to join their team.In this role, you'll lead the production of user-centred software and work with multi-disciplinary te... Show more

 • Promoted

Principal Engineer: AI-Driven Workflow Orchestration

ExperisKnutsford, England, United Kingdom
Full-time

We are a leading banking institution seeking a principal engineer to drive evaluation and architecture for workflow orchestration using Camunda or Temporal.The role focuses on scalable patterns, en... Show more

 • Promoted

Senior Python Engineer — Cloud & Microservices Leadership

NCC Group plcManchester, England, GB
Full-time

NCC Group plc is looking for a Senior Python Developer in Manchester.This role focuses on designing and delivering high-quality Cyber Security-related software solutions.Candidates should have stro... Show more

 • Promoted

Remote Full-Stack Engineer — AI-Driven Python & React

Version 1Manchester, England, GB
Remote
Full-time

A leading technology service provider in Manchester is looking for a software developer to join their Scrum Team.You will contribute to technology decisions and work on AI-driven initiatives using ... Show more

 • Promoted

Senior Python Developer

Peaple TalentWest of England, England, GB
Permanent +1

Peaple Talent have partnered with a phenomenal tech disrupter based in Bristol who are looking to grow their team with an experienced contract Python Developer to support in the next iteration of t... Show more

 • Promoted

Senior/ Lead Applied AI Engineer - client Facing - Exciting Scaling AI Scaleup with significant[...]

Opus Recruitment SolutionsManchester, England, United Kingdom
Full-time

Senior/ Lead Applied AI Engineer - client Facing - Exciting Scaling AI Scaleup with significant investment - up to £135k!.Opus is partnering with an innovative AI consultancy that is helping leadin... Show more

 • Promoted

AI Engineer Starter Course (Blackburn)

ITonlinelearning RecruitmentBlackburn, ENG, GB
Full-time

AI Trainee Engineer – £45,000–£75,000 – Job Programme.This is a self-funded programme that leads to employment, fees apply.Complete the programme and get a job, or get... Show more

 • Promoted

Senior Full-Stack AI Engineer — Hybrid

Fitch GroupManchester, England, GB
Full-time

Fitch Group in Manchester is seeking a Senior Software Engineer, Full Stack - AI, to design innovative ML/AI solutions.In this role, you'll collaborate with agile teams, utilizing a modern technolo... Show more

 • Promoted

Senior Full-Stack AI Engineer (Remote, React/Python)

Version 1Manchester, England, GB
Remote
Full-time

A leading technology services provider in Manchester seeks a skilled developer to work in an Agile Scrum Team.The role emphasizes front-end development with React and knowledge of Python for buildi... Show more

 • Promoted

Senior Software Engineer — AI/ML Backend & Data Pipelines

DatalexManchester, England, GB
Full-time

Datalex is seeking a Senior Software Engineer specializing in Python-based AI/ML development to design and deploy machine learning features.The role involves building APIs, collaborating on CI/CD, ... Show more

 • Promoted

Senior DevOps Engineer Python AWS CDK

Client ServerManchester, England, GB
Full-time

Get AI-powered advice on this job and more exclusive features.This range is provided by Client Server.Your actual pay will be based on your skills and experience — talk with your recruiter to learn... Show more

 • Promoted

Senior Data Engineer - AI-Driven Data Pipelines

慨正橡扯Manchester, England, GB
Full-time

Moody’s is seeking a data engineer in Manchester to design, build, and support highly scalable data pipelines powering Moody’s next‑generation digital content platform.You will collaborate with tec... Show more

 • Promoted

Senior ML Engineer (Python, GenAI) - Part-time Project-Based

MindriftManchester, England, United Kingdom
Part-time +1

Please submit your CV in English and indicate your level of English proficiency.Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, eva... Show more

 • Promoted

Senior Python Developer - AI SaaS, AWS Microservices

Peaple TalentWest of England, England, GB
Permanent

A tech disrupter in the UK is seeking an experienced contract Python Developer to enhance their SaaS platform.You will be responsible for building cloud-based microservices and working closely with... Show more

 • Promoted

Senior AI Architect — Python & Multi-Agent Systems

DeliveryAppManchester, England, GB
Full-time

A leading logistics technology company in Manchester is looking for an experienced Senior Python Developer.In this role, you will take ownership of designing and deploying advanced AI systems, tack... Show more

 • Promoted

Senior Python Developer - Agentic AI Systems

ZippdManchester, England, GB
Full-time

DeliveryApp is in an exciting phase of rapid growth.As we scale our operations and technology, we are expanding our engineering team to build the next generation of our platform.We are looking for ... Show more

 • Promoted

Remote Python Backend Engineer — AI-Driven APIs

YO IT ConsultingManchester, England, GB
Remote
Full-time

YO IT Consulting is seeking a Python Backend Developer for a contract role focused on building scalable REST and GraphQL services in a remote setting.You will work with data validation, API securit... Show more