Data Engineer

We are currently recruiting a Data Engineer for one of our clients. The role is outside IR35 and is paying £400-500 per day, it will initially be for 6 months. It is also fully remote.

Key Responsibilities

• Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark)
• Build and optimize ETL / ELT workflows for large-scale structured and unstructured data
• Implement Delta Lake architectures (Bronze / Silver / Gold layers)
• Integrate data from multiple sources (databases, APIs, event streams, files)
• Optimize Spark jobs for performance, scalability, and cost
• Manage data quality, validation, and monitoring
• Collaborate with analytics and ML teams to support reporting and model development
• Implement CI / CD, version control, and automated testing for data pipelines

Required Qualifications

o AWS : S3, EMR, Glue, Redshift, Lambda, IAM

o Azure : ADLS Gen2, Azure Databricks, Synapse, Data Factory, Key Vault