Python/Spark/AI Developer

Our client is seeking an experienced Python / Spark / AI Developer to help replatform legacy T-SQL workloads onto modern Spark and Delta Lake pipelines while developing robust APIs and AI-enabled capabilities. This intermediate-to-senior role is suited to a developer who is comfortable working autonomously in a specification-driven environment and delivering production-quality, well-tested code.

Key Responsibilities

• Build production-grade Spark/PySpark pipelines using Delta Lake.

• Replatform legacy T-SQL logic into Spark SQL while preserving functional parity.

• Develop modern, type-safe Python using strong testing and static-analysis practices.

• Build REST APIs using FastAPI or equivalent frameworks.

• Implement authentication, idempotency and reliable job-status semantics.

• Validate migrated pipelines against legacy systems and provide evidence of output parity.

• Work from design documents and Architecture Decision Records (ADRs).

• Contribute tests and documentation alongside code changes.

• Participate in Git-based, PR-driven development and CI processes.

Essential Requirements

• 4+ years' professional Python development experience.

• 2+ years' production experience building Spark/PySpark or comparable distributed data pipelines.

• Strong Python 3.12 experience, including type hints, Pydantic, modern packaging and static analysis.

• Strong Apache Spark/PySpark knowledge, including DataFrame API, Spark SQL, partitioning and performance tuning.

• Understanding of Spark driver/executor architecture and Spark Connect.

• Production experience with Delta Lake or equivalent lakehouse technologies such as Iceberg or Hudi.

• Strong SQL skills and the ability to translate legacy T-SQL logic into Spark SQL.

• Experience with MERGE operations, schema evolution, idempotent writes, hashing, surrogate keys and deduplication.

• Strong automated testing experience using pytest.

• REST API development experience, preferably with FastAPI.

• Experience with Docker-based development environments.

• Strong Git and CI/CD discipline, including linting, type-checking and automated testing.

• Bachelor's degree in Computer Science, Engineering or equivalent experience.

• Ability to work independently from written specifications and technical requirements.

Advantageous

• LLM integration and AI feature development.

• Experience with Ollama, vLLM, llama.cpp, LM Studio or hosted AI APIs.

• Data governance, PII protection and privacy engineering.

• Experience with Hive Metastore, Trino or Apache Ranger.

• Azure data technologies including ADLS Gen2, Synapse/ADF, Key Vault or Service Bus.

• Observability tools such as OpenTelemetry, OpenLineage or structured logging.

• Data migration and output-parity experience.

• Databricks Certified Developer for Apache Spark or equivalent certification.

Only shortlisted candidates will be contacted.

Submit your CV and transcripts to it.jobs@tumaini.co.za or call 031 350 4018.

Visit our website for more exciting career opportunities: https://tumaini.co/

Correspondence will only be conducted with short listed candidates. Should you not hear from us within 4 days, please consider your application unsuccessful.