Intermediate Data Engineer

Key Responsibilities

Data Engineering & Platform Development

• Design, build, and maintain scalable data pipelines and workflows.
• Develop and optimise data processing solutions using Databricks and Apache Spark .
• Build and support end-to-end data integration and transformation processes.
• Ensure data solutions are reliable, efficient, and aligned with best-practice engineering standards.

Cloud & Big Data Solutions

• Develop solutions within Azure cloud environments.
• Support the implementation and optimisation of modern data architectures.
• Work with large-scale datasets and distributed computing frameworks.
• Contribute to the ongoing evolution of the organisation's data platform capabilities.

Data Pipeline Automation & CI/CD

• Develop and maintain CI/CD pipelines to support efficient deployment and release processes.
• Implement automation strategies that improve data delivery, quality, and operational efficiency.
• Support monitoring, testing, and optimisation of data engineering workflows.

Collaboration & Solution Delivery

• Work closely with Data Engineers, Developers, Analysts, and Business Stakeholders to deliver high-quality data solutions.
• Participate in solution design discussions and technical planning.
• Contribute to the continuous improvement of data engineering practices and standards.

Innovation & Continuous Learning

• Stay current with emerging technologies and trends within data engineering and cloud platforms.
• Identify opportunities to improve performance, scalability, and maintainability across data solutions.
• Contribute to knowledge sharing and technical excellence within the team.

Requirements

Essential Skills & Experience

• 3-5 years' experience in a Data Engineering, Backend Development, or similar technical role.
• Previous experience working as a Data Engineer .
• Strong experience with Databricks .
• Proven expertise in Apache Spark and cluster computing environments.
• Advanced Python skills, including mandatory PySpark experience.
• Strong SQL development skills.
• Experience developing and maintaining CI/CD pipelines .
• Experience building scalable and reusable data pipelines.
• Solid software engineering and backend development foundations.
• Exposure to Azure cloud services and infrastructure .

Advantageous Experience

• Microsoft Fabric experience.
• Medallion Architecture experience.
• Experience working with modern data warehousing and analytics environments.
• Exposure to cloud-native

.special-hidden { display: none; }