Python/Spark/AI Developer

Location: Hybrid / Remote Employment Type: Full-Time Industry: Data Engineering | AI | Software Development | Data Platforms

WatersEdge Solutions is partnering with a number of leading South African and international organisations to recruit an experienced Python/Spark/AI Developer to join a technically driven team working on the modernisation of large-scale data platforms.

This role will focus on replatforming legacy T-SQL workloads into modern Spark and Delta Lake pipelines , while building the APIs and AI capabilities that sit around them. You’ll work primarily in Python, with a strong emphasis on type-safe, tested and spec-driven development.

This is an excellent opportunity for an intermediate-to-senior developer who enjoys solving complex data engineering problems and wants to work across Python, distributed data processing, lakehouse architecture, APIs and emerging AI technologies .

About the Role

As Python/Spark/AI Developer, you’ll play a key role in migrating legacy data workloads onto a modern Spark-based architecture.

You’ll translate existing T-SQL logic into Spark SQL and PySpark pipelines, validate migrated workloads against legacy outputs, and ensure the new platform delivers reliable and demonstrable parity.

The engineering environment is heavily spec-driven. You’ll work autonomously from written requirements, design documentation and Architecture Decision Records (ADRs), with changes expected to include appropriate testing and documentation rather than code alone.

Alongside the core data engineering work, you’ll build REST APIs around pipeline jobs and contribute to AI capabilities, including provider-neutral LLM integrations and structured AI outputs.

Key Responsibilities

Build production-grade Spark and PySpark pipelines on Delta Lake.

Replatform legacy T-SQL workloads and stored-procedure-era logic into Spark SQL.

Translate existing data logic accurately while maintaining expected business and data semantics.

Write modern, type-hinted Python with strict static analysis.

Develop and maintain automated pytest suites covering unit, integration and end-to-end testing.

Validate migrated pipelines against legacy systems and provide clear parity and regression evidence.

Build REST APIs using FastAPI or equivalent frameworks.

Implement API request validation, authentication, idempotency and job-status semantics.

Work within Docker-based development environments using Compose stacks.

Follow GitHub flow and PR-driven development practices.

Maintain linting, type-checking and automated test gates within CI pipelines.

Work from des

.special-hidden { display: none; }