Permanent

Data Engineer - Azure Databricks

cLogo
83Zero Ltd

Job Description

Location: London - Hybrid (3 days per week in the office)

Salary: £90,000 - £100,000 + Benefits

We are looking for an experienced Senior Data Engineer to join a major organisation working on a large-scale Azure Databricks platform.

This is a hands-on technical role focused on building, developing and optimising complex data pipelines, improving data quality and supporting the ongoing development of a modern cloud-based data platform.

The successful candidate will have extensive experience in data engineering, strong Azure expertise and a proven background working with Databricks in complex enterprise environments.

Key Responsibilities

  • Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and SQL.
  • Build and optimise ETL/ELT processes, integrating data from APIs, databases and other external sources.
  • Work extensively with Databricks Unity Catalog, Delta Lake and Spark SQL.
  • Develop complex data transformations, ensuring data accuracy, consistency and performance.
  • Implement data quality checks, monitoring and validation processes.
  • Utilise Microsoft Purview for data governance, lineage tracking and metadata management.
  • Optimise Databricks workloads for performance, scalability and cost efficiency.
  • Develop and maintain CI/CD pipelines using Azure DevOps and other automation tools.
  • Collaborate ..... full job details .....