Python / Spark / AI Developer

Job Description

• We are looking for a skilled Python / Spark / AI Developer to join a growing technology team.
• You will be responsible for developing scalable data pipelines, replatforming legacy SQL workloads onto modern Spark and lakehouse technologies, and supporting the development of APIs and AI-driven features.
• The ideal candidate is a strong Python developer with hands-on production experience in Apache Spark/PySpark, SQL, Delta Lake and REST APIs.

Key Responsibilities

• Build Spark/PySpark data pipelines using Delta Lake.
• Replatform legacy T-SQL logic into Spark SQL.
• Develop modern, type-safe and well-tested Python applications.
• Build REST APIs using FastAPI or equivalent frameworks.
• Implement authentication, idempotency and job-status functionality.
• Validate migrated pipelines against legacy systems and provide parity evidence.
• Work from technical specifications, design documents and ADRs.
• Develop automated tests using pytest.
• Contribute to CI/CD pipelines, code reviews and technical documentation.

Minimum Requirements

• 4+ years of professional Python development experience.
• 2+ years of production experience with Apache Spark / PySpark.
• Strong knowledge of Spark SQL and DataFrame APIs.
• Production experience with Delta Lake or equivalent technologies such as Iceberg or Hudi.
• Strong SQL skills and experience working with legacy T-SQL.
• Experience with pytest and automated testing.
• Experience developing REST APIs, preferably with FastAPI.
• Experience with Docker and containerised development environments.
• Strong Git and CI/CD experience.
• Experience with type-hinted Python and static analysis tools such as Pyright or Mypy.
• Experience with modern Python tooling such as uv or Poetry.
• Ability to work independently from written technical requirements and design documentation.
• Bachelor's degree in Computer Science, Engineering or a related field, or equivalent experience.
• Proven track record of delivering production Python/Spark pipelines independently.

Advantageous Skills

• Experience with LLM / AI integration.
• Experience with Ollama, vLLM, llama.cpp, LM Studio or hosted AI APIs.
• Knowledge of data governance, privacy engineering and POPIA/GDPR principles.
• Experience with Hive Metastore, Trino or Apache Ranger.
• Azure experience, including ADLS Gen2, Synapse, ADF, Key Vault or Service Bus.
• Experience with structured logging, OpenTelemetry or OpenLineage.
• Data migration and pipeline parity testing experience.
• Databricks Certified Developer for Apache Spark or equivalent certification.