Job Description
Location: London - Hybrid (3 days per week in the office)
Salary: £90,000 - £100,000 + Benefits
We are looking for an experienced Senior Data Engineer to join a major organisation working on a large-scale Azure Databricks platform.
This is a hands-on technical role focused on building, developing and optimising complex data pipelines, improving data quality and supporting the ongoing development of a modern cloud-based data platform.
The successful candidate will have extensive experience in data engineering, strong Azure expertise and a proven background working with Databricks in complex enterprise environments.
Key Responsibilities
- Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and SQL.
- Build and optimise ETL/ELT processes, integrating data from APIs, databases and other external sources.
- Work extensively with Databricks Unity Catalog, Delta Lake and Spark SQL.
- Develop complex data transformations, ensuring data accuracy, consistency and performance.
- Implement data quality checks, monitoring and validation processes.
- Utilise Microsoft Purview for data governance, lineage tracking and metadata management.
- Optimise Databricks workloads for performance, scalability and cost efficiency.
- Develop and maintain CI/CD pipelines using Azure DevOps and other automation tools.
- Collaborate ..... full job details .....