Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • London, England, GB
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • London, England, GB
26 days ago
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Create a job alert for this search

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • London, England, GB

Similar jobs

Senior Python Engineer

Grid Dynamics HoldingsGreater London, England, GB
Full-time

We are seeking experienced engineers to join our client projects.You’ll be responsible for engineering global scale software systems in banking, hedge funds and other financial institutions.We’re l... Show more

 • Promoted

Staff Machine Learning Engineer (£220,000 | RecSys | Python)

DexGreater London, England, United Kingdom
Full-time

London, 4 days in office (Shoreditch).Up to £220,000 + Bonus + Compelling Equity.This role is with one of Dex’s trusted partner companies.We work closely with their teams to truly understand their ... Show more

 • Promoted

Senior Python Backend Engineer for AI Prototypes (Remote)

Olive Jar DigitalGreater London, England, GB
Remote
Full-time

A growing digital consultancy in the UK is seeking a Senior Software Developer (Python) to design and enhance AI prototypes.This role involves working collaboratively with engineers, managers, and ... Show more

 • Promoted

Senior AI Platform Engineer for Enterprise-scale Impact

WriterGreater London, England, GB
Full-time

Writer is seeking a Senior Software Engineer to empower enterprises in adopting AI-powered solutions.Your role will involve making architectural decisions and developing secure, scalable systems th... Show more

 • Promoted

Lead Software Engineer - Python / AI

J.P. MORGANGreater London, England, United Kingdom
Full-time

Join us to shape the future of equity derivatives trading through technology.You'll partner closely with trading desks, technology teams, and Quantitative Research to deliver impactful solutions.We... Show more

 • Promoted

Senior AI Software Engineer: Coding Agent for Game Dev

Lightspeed StudiosGreater London, England, GB
Full-time

Lightspeed Studios is looking for a Sr.AI Software Engineer to lead the development of an innovative in-house AI Coding Agent.This role will significantly enhance our game studios' development pipe... Show more

 • Promoted

Senior MLOps Engineer: AWS AI Platform (On-site London)

Talenzon groupGreater London, England, United Kingdom
Temporary

Talenzon group is seeking a Senior MLOps Engineer to join our London-based on-site team for a 6-month contract.You will design and implement an enterprise MLOps platform on AWS, deploy production-r... Show more

 • Promoted

Senior Python Engineer – Data Systems & AI Platform

ParkopediaGreater London, England, GB
Full-time

Parkopedia, a leading connected car services provider, seeks a Staff Python Engineer to lead data‑science products within Arrive.You will design scalable data services, influence architecture, and ... Show more

 • Promoted

Senior Python Engineer

Tickerface LtdGreater London, England, GB
Full-time

Tickerface is building the world's first shoppable content monetization platform.We believe shopping should be a fun and interactive experience.Our product lets content publishers enable their audi... Show more

 • Promoted

Deals Technology, AI Forward Deployed Engineer – Senior Associate

JobtailorGreater London, England, GB
Full-time

AI solutions, tools, and platforms. Show more

 • Promoted

Senior AI Software Engineer: Coding Agent for Game Dev

TencentGreater London, England, GB
Full-time

A leading tech company in Greater London is seeking a Sr.AI Software Engineer to architect an innovative AI Coding Agent.This role involves leading the development of a tool that accelerates game s... Show more

 • Promoted

Senior Python Engineer - AWS and AI

TalentHawkGreater London, England, GB
Full-time

Senior Python Engineer - AWS and AI - 2 days a week in London.Our client is a market‑leading fintech platform looking for a.In this role, you will design scalable microservices handling massive fin... Show more

 • Promoted

Senior AI Engineer - Lead AI Agent Platform (Remote)

Story Terrace Inc.Greater London, England, United Kingdom
Remote
Full-time

Senior AI Engineer to own and evolve the AI agent at the heart of our bid-management platform.You’ll work across Python backend services and React/TypeScript interfaces to ship reliable features fo... Show more

 • Promoted

Senior AI Engineer (Python) – Hybrid, London

Market Cloud LtdGreater London, England, GB
Full-time

A technology solutions provider is looking for an AI Engineer with robust Python expertise to help build intelligent, scalable AI solutions.The role requires developing, training, and deploying ML/... Show more

 • Promoted

Senior Lead Software Engineer: Python, Cloud & AI/ML

FairygodbossGreater London, England, GB
Full-time

Fairygodboss is looking for a Senior Lead Software Engineer to join the Markets Research Technology team at JPMorgan Chase in London.This role requires a seasoned member of an agile team to design ... Show more

 • Promoted

Senior AI Engineer

ITH PharmaGreater London, England, GB
Full-time

Salary: £75,000 - 85,000 per year.Strong experience in AI engineering, ML/LLM applications, or intelligent automation.Solid Java engineering background with production-quality coding practices.Expe... Show more

 • Promoted

Senior Python Engineer

Hunter BondGreater London, England, United Kingdom
Full-time

Salary: Up To £130,000 P/A + Comps.A high-performing global team is looking for an experienced Python Engineer to help build and enhance critical backend platforms supporting complex trading, risk ... Show more

 • Promoted

Senior Python AI Engineer - GenAI & LLM Solutions (Hybrid)

SaragossaGreater London, England, GB
Full-time

Saragossa, a fast-growing AI consultancy in the UK, seeks a capable Python software developer to build AI solutions, including LLM/GenAI products, with strong data and microservices skills.You will... Show more

 • Promoted

Senior AI Operations Engineer (Python/Golang)

Sky Ireland LimitedGreater London, England, GB
Full-time

A leading media company is seeking a Senior Software Engineer in Greater London to design, build, and maintain complex systems for AI-assisted operations.The role requires proficiency in Golang and... Show more

 • Promoted

Senior Python/Go Engineer – Terraform & ML (Contract)

Golang WorksGreater London, England, GB
Full-time

A software development company in London is looking for Senior Software Developers/Engineers skilled in Python, Terraform, and Go.Ideal candidates will have 7 to 10 years of experience and proficie... Show more