Lead AI Engineer - Permanent

Job Description

This role requires once in a week travel to Newcastle.

About Scrumconnect Consulting

Scrumconnect Consulting is a multi-award-winning digital consultancy, recognised for delivering impactful and innovative technology solutions across UK government departments. Our work has positively influenced the lives of over 40 million UK citizens. We are passionate about user-centred design, agile delivery, and building digital services that make a real difference — and we're now scaling that expertise into large, high-stakes AI adoption programmes across the public sector.

Overview

We're looking for a Lead AI Engineer to be the hands-on technical builder at the core of a large-scale AI Operating Model programme for a central government department. Where the Lead Technical Architect sets direction, you turn it into working, production-grade systems — semantic search, RAG pipelines, and broader generative AI capability — integrated into complex legacy and multi-cloud environments handling high-volume, sensitive public sector data.

You'll work inside a collaborative "Rainbow Team" alongside civil servants and the wider delivery team, staying close to the code while also mentoring engineers and helping build the internal capability the department needs to eventually run these systems without long-term reliance on external suppliers.

Key ResponsibilitiesHands-On Build & Delivery Design, build, and ship production AI components — RAG pipelines, retrieval and embedding infrastructure, orchestration logic, and integration layers — writing high-quality, tested, maintainable code and staying close to implementation rather than delegating it away.

Technical Leadership Within the Squad Lead the engineering practice within your delivery team: set coding standards, review designs and pull requests, unblock technically complex problems, and mentor other engineers day to day.

Responsible AI in Practice Implement the guardrails the Architect designs — bias mitigation checks, evaluation harnesses, human-in-the-loop review points — so that Responsible AI principles (ATRS alignment, NCSC "Secure by Design, " meaningful human control) are enforced in the running system, not just on paper.

Reliability, Security & Observability Build AI services to be secure-by-default and observable in production: logging, monitoring, alerting, and rollback paths appropriate for sensitive public-sector data and high-availability requirements.

Knowledge Transfer & Capability Uplift Work within the OKUA (Ownership, Knowledge, Understanding, Awareness) framework and "Docs-as-Code" practices to pair with and upskill internal government engineers, so capability genuinely transfers rather than staying locked in the consultancy team.

Efficient, Sustainable Engineering Favour low-modality, resource-efficient designs where they meet the need — right-sizing models and infrastructure rather than defaulting to the largest/most expensive option — in line with the programme's Green AI and Net Zero commitments.

Required Skills & Experience

Loosely mapped to the Government Digital and Data (DDaT) framework, at Lead Engineer level:

● Coding and Scripting (Expert) — writing production-grade, well-tested code; setting standards for others; comfortable owning components end-to-end.

● Systems Design (Practitioner) — designing components that integrate cleanly into a wider, architect-defined system; understanding trade-offs across the stack.

● Data Engineering (Practitioner) — building reliable pipelines to ingest, clean, and prepare data (including unstructured/legacy sources) for AI consumption.

● DevOps / Continuous Delivery (Practitioner) — CI/CD pipelines, infrastructure-as-code, and automated deployment for AI workloads specifically (not just conventional web services).

● Testing & Evaluation (Practitioner) — beyond conventional unit/integration testing, building evaluation harnesses for AI system quality: retrieval accuracy, hallucination rate, bias/fairness checks.

● Problem Solving (Practitioner) — diagnosing and resolving complex, ambiguous technical issues under production pressure.

● Agile Working (Practitioner) — delivering iteratively within a blended, multidisciplinary team including civil servants.

Given the scale of this programme, direct hands-on depth across the AI engineering stack is essential:

●Strong general-purpose programming (most commonly Python) applied to AI/ML systems

● Semantic search, vector/embedding infrastructure, and RAG pipeline construction

● LLM orchestration and agentic frameworks (e.g. LangChain/LlamaIndex-style tooling, multi-agent patterns)

● Prompt engineering and systematic evaluation/guardrail tooling (hallucination detection, safety testing)

● LLMOps/MLOps — model versioning, deployment, monitoring, and rollback for AI services in production

● Cloud-native engineering across major providers (AWS, Azure, or GCP), including their AI/ML tooling

● Containerisation and infrastructure-as-code for repeatable, auditable deployments

● Secure-by-design engineering appropriate to sensitive public-sector data

Desirable Experience

● AWS / Azure / GCP certifications (associate or professional level)

● Prior delivery of AI or digital services within UK central government or wider public sector

● Experience fine-tuning or adapting open-source/foundation models for a specific domain

● Open-source contributions or active engagement in AI/ML engineering communities

● Experience designing for sustainability/Green IT commitments

●The technological capabilities can span a wide range of AI and automation tools, including Virtual Assistants, Robotic Process Automation (RPA), Computer Vision, Natural Language Processing (NLP), Machine Learning, Deep Learning, Generative AI, Frontier LLMs, and Predictive AI. A strong candidate would have experience/exposure to nuances of these tools/technologies.

Location and Working Pattern

Hybrid, based from Newcastle upon Tyne, with on-site attendance required for workshops, knowledge-transfer sessions, and onboarding. All production system and data access must be performed solely from within the UK.

How We Work

Our values:

● Collaboration — we are stronger as a team than as individuals; we tackle problems and celebrate wins together.

● Create Value Early — we find the quickest route from idea to product, because our clients rely on us to do what's best.

● Integrity — teamwork requires trust; we can always be relied upon to uphold the highest standards.

● Commitment — no matter what, no matter how, we give our 100% to keeping our promises and achieving our goals.

● Diversity and Inclusion — we believe diversity drives innovation and better outcomes, and we're committed to an inclusive environment where every individual is valued, respected, and supported. We welcome applications from candidates of all backgrounds, including women, people with disabilities, and diverse communities, as well as those seeking flexible working arrangements.

As a Disability Confident Level 1 employer, we provide reasonable adjustments throughout recruitment and employment to ensure equal opportunity for all.

How to Apply

Please submit your CV. Shortlisted candidates will be invited to interview. This role requires BPSS (Baseline Personnel Security Standard) clearance — please flag in your application if you already hold clearance.

Requirements
What we are looking for: The interview will consist of both behavioural questions which follow the STAR methodology based on your past experience (see interview tips below) as well as detailed technical interview across our main technology stack described below. Must have: - Proven hands on track record experience managing large cloud based infrastructure deployments via IaC on Azure via terraform. - Proven hands on track record experience managing VM deployments on Azure and via Ansible. - Proven hands on track record experience implementing CI / CD pipelines to build, package, deploy, test and promote to higher environments infrastructure as well as applications, preferably via ADO. - Detailed understanding of infrastructure main building blocks and design, how they all fit together and how to manage them via code. That includes Virtual Networks, Routing, DNS, NSGs, Identity Management, Firewall Setup, Security Policies, Secrets and Certificates management. - Experience with configuring SecOps tools, preferably SonarQube, Checkmarx, PMD and using pre commit hooks across git based repositories to support shift left approach on the organisation deliverables. - Ability to resolve complex problems and work collaboratively with multiple technical and non technical teams with different skill sets and backgrounds. - Last but certainly not least: Excellent verbal and written communication skills. Nice to have: - Experience with Linux and Windows System administration including OS patching, permissions management, security settings, ssh / RDP enablement, using debugging and tracing tools to triage common problems. - Experience with at least one market leading monitoring tool, understand how it operates, configured and managed.