Data Engineer

Senior Data Engineer – Spark & Databricks Platform Build

Location : London (2 days a week on site)

Contract : 6-month sign-on

Interview process : 2 stages

twentyAI’s customer is building the next generation of their data platform, with Databricks and Apache Spark at the core. A PoC is already in place — now they’re looking for someone who’s done this before to lead the full-scale build and help roll the platform out across the organisation.

The project involves migrating the existing on-prem Data Lake to a Databricks-based architecture, while continuing to leverage private cloud storage. You’ll be laying the foundations : building ETL pipelines in PySpark, implementing platform best practices, and working closely with teams across trading and finance.

Profile :

• Strong experience delivering Spark / Databricks implementations in a lead or senior role
• Solid hands-on background in PySpark, Spark, and Python
• Experience setting up data platforms with a DevOps-first approach (IaC, CI / CD, automation)
• Exposure to AWS-based environments
• Familiarity with financial / trading systems and working in regulated industries (e.g., banking, commodities)

Tech Environment :

J-18808-Ljbffr