Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • Birmingham, England, GB
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • Birmingham, England, GB
25 days ago
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Create a job alert for this search

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • Birmingham, England, GB

Similar jobs

AI Platform Engineer: Agentic LLMs & Production

Goldman Sachs Group, Inc.Birmingham, England, GB
Full-time

Asset & Wealth Management Software Engineer in Birmingham to lead AI projects that enhance program delivery.The role demands a strong background in software development, particularly in Python and ... Show more

 • Promoted

Senior Data Platform Engineer — Cloud Data Lakes & AI

CapgeminiBirmingham, England, GB
Full-time

A leading global consulting firm is seeking a Data Engineer in Birmingham to design and implement high-performance data pipelines and maintain secure data platforms.The role requires at least 10 ye... Show more

 • Promoted

Senior Data Pipeline Engineer - Backend & AI Systems

Obsidian SecurityCheltenham, England, GB
Full-time

Obsidian Security seeks a Software Engineer to enhance their AI and supply chain risk products.In this role, you will design and build data pipelines, implement APIs, and collaborate with teams in ... Show more

 • Promoted

Hybrid Senior Data Engineer - Climate SaaS (Python, PySpark)

Revolution TechnologyWest Midlands Combined Authority, England, GB
Full-time

A product-led B2B SaaS company in the West Midlands is seeking a Senior Data Engineer to lead in revamping their platform, focusing on sustainable solutions.The ideal candidate will have strong ski... Show more

 • Promoted

Python Software Engineer

TEAMBirmingham, England, GB
Full-time

Salary: £65,000 - 74,000 per year.Commercial experience developing applications with Python and frameworks such as Django alongside JavaScript or TypeScript.Experience working within cloud environm... Show more

 • Promoted

Cloud Engineer

AccentureCheltenham, England, GB
Full-time

Salary: Competitive salary and package dependent on experience + shift work.Career Level: We are hiring at levels - Analyst & Senior Analyst.Please Note: Any offer of employment is subject to satis... Show more

 • Promoted

Senior LEV Engineer

Extra Recruitment LimitedStafford, ENG, GB
Full-time

Extra Recruitment are recruiting for a Senior LEV Engineer in the Stafford area.Details of a Senior LEV Engineer.Lead and conduct Thorough Examination and Testing (TExT) of LEV systems in accordanc... Show more

 • Promoted

Software Engineer - AI in Stafford - Agena Group

WorksHubStafford, England, GB
Full-time

Software Engineer - DAML Application Runtime (UK).Staff-Level Full-Stack Software Engineer.E 16th Street, New York, NY 10003.Design, develop, and maintain robust software solutions using object-ori... Show more

 • Promoted

Senior AI QA Automation Engineer | CI/CD & Python

AutodeskBirmingham, England, GB
Full-time

Autodesk is seeking an experienced Senior QA Automation Engineer for AI Applications UK in Birmingham to ensure quality, reliability, and performance of AI-powered applications.You will design scal... Show more

 • Promoted • New!

Senior Full-Stack AI Engineer: Build Production-Ready AI

AmberesBirmingham, England, GB
Full-time

A leading technology company is seeking a Full Stack Engineer to join their team in Birmingham.The ideal candidate will have 5+ years of experience in software engineering and be proficient in Reac... Show more

 • Promoted

Senior AI Solutions Engineer - Python, FastAPI, Azure

Gateley PlcBirmingham, England, GB
Full-time

A leading legal and professional services group in Birmingham is seeking an experienced AI Developer to design and deploy AI-powered applications.The role involves building scalable AI services usi... Show more

 • Promoted

Senior Python Developer

NCC Group plcCheltenham, England, GB
Full-time

Cyber Services and Capabilities.An exciting opportunity to be one of the core team members of the Software Engineering team within NCC Group’s GMS division.You will be a development SME playing a k... Show more

 • Promoted

Python Data Engineer

ApexonBirmingham, England, GB
Full-time

Apexon is a digital-first technology services firm specializing in accelerating business transformation and delivering human-centric digital experiences.We have been meeting customers wherever they... Show more

 • Promoted

Python Software Engineer - £75,000 A Year

SF RecruitmentCheltenham, United Kingdom Of Great Britain And Northern Ireland, UK
Full-time

Python Software Engineer with strong container orchestration and a passion for machine learning is sought by a high growth scale up based near Cheltenham.Working at the forefront of defence and nat... Show more

 • Promoted

Senior Software Engineer - Conversational AI in Healthcare

Transform9Birmingham, England, GB
Full-time

A healthcare technology company in Birmingham is looking for a Senior Software Engineer to enhance their innovative conversational agent platform.The role involves designing and implementing scalab... Show more

 • Promoted

Full Stack Engineer (Ai Startup) - €100.000 - €160.000 A Year - Remote

Oliver BernardCheltenham, United Kingdom Of Great Britain And Northern Ireland, UK
Remote
Full-time

Full Stack Engineer (AI & Cybersecurity Startup) - EU Remote.We're partnering with a well-funded AI startup (stealth for now) that's building at the intersection of LLMs and cybersecurity, and we'r... Show more

 • Promoted

Senior Python Engineer | Lead & Mentor Team

Noriker Power LtdCheltenham, England, GB
Full-time

Noriker Power Ltd is seeking an experienced software developer to support business objectives through the development of quality software and web applications.The ideal candidate will have at least... Show more

 • Promoted

Python Software Engineer

Third Nexus Group LimitedBirmingham, England, GB
Permanent

Salary: £60,000 - 74,000 per year.Hands-on experience in Python and a web development framework such as Django, plus JavaScript or TypeScript, ideally in cloud environments such as AWS, Azure or GC... Show more

 • Promoted

Python Automation Engineer

Auxo TalentBirmingham, England, GB
Full-time

Birmingham 2-3 days p/w, remaining remote.We are looking for an experienced Python Automation Engineer to join a professional services team delivering enterprise data automation solutions for large... Show more

 • Promoted

Senior Forward-Deployed AI Engineer | Client-Facing

EYBirmingham, England, GB
Full-time

EY seeks an Associate Director in Forward Deployed Engineering to design, develop, and deploy AI-driven, data-centric solutions for strategic clients.You will blend deep technical skills with leade... Show more