Confluent Kafka Migration Engineer

Role Overview

Our client is looking for experienced (mid-level Senior) Confluent Kafka Migration Engineers to support the migration of a large-scale, on-premises Confluent Platform environment to Confluent Cloud. The resource will be responsible for migration planning, cloud environment configuration, secure network connectivity, Kafka data replication, authentication modernisation, production cutovers, and post-migration stabilisation.

The role requires strong hands-on expertise in Apache Kafka, Confluent Platform, Confluent Cloud, and enterprise-scale production migrations, with a focus on data integrity, high availability, security, and minimal service interruption.

Roles and Responsibilities

• Assess the existing on-premises Kafka and Confluent Platform environment and validate the target cloud architecture.

• Develop detailed migration strategies, wave-based execution plans, cutover runbooks, rollback procedures, and risk mitigation plans.

• Provision and configure Confluent Cloud environments for development, pre-production, and production.

• Configure Kafka topics, partitions, replication, Schema Registry subjects, ACLs, service accounts, and Kafka Connect connectors.

• Implement and manage Confluent Cluster Linking for replication and migration of Kafka workloads from on-premises to cloud.

• Validate data consistency, schema compatibility, consumer offsets, application connectivity, and message processing before and after migration.

• Configure secure private connectivity between on-premises data centres and cloud environments using AWS PrivateLink, GCP Private Service Connect, VPC peering, or equivalent technologies.

• Modernize Kafka client authentication by migrating from Kerberos-based authentication to supported mechanisms such as API keys, OAuth, or mutual TLS (mTLS), as applicable to the target architecture.

• Collaborate with application teams to update Kafka client configurations, libraries, and authentication settings.

• Implement Infrastructure as Code (IaC) using Terraform and automate platform provisioning and configuration through CI/CD pipelines.

• Configure monitoring, alerting, audit logging, consumer lag tracking, and performance dashboards using enterprise observability tools.

• Troubleshoot Kafka replication, connectivity, throughput, partitioning, consumer lag, and production performance issues.

• Execute production cutovers in planned waves, validate workload functionality, and maintain rollback readiness.

• Support hypercare, operational handover, knowledge transfer, and decommissioning of legacy Kafka infrastructure after successful migration.

Required skills

• Strong hands-on experience with Apache Kafka and Confluent Platform administration.

• Proven experience in enterprise-scale Kafka migrations from on-premises environments to cloud-managed platforms.

• Practical experience with Confluent Cloud architecture, configuration, administration, and operations.

• Hands-on experience with Confluent Cluster Linking and Kafka replication strategies.

• Strong understanding of Kafka topics, partitions, replication factors, consumer groups, offsets, and high-availability configurations.

• Experience with Confluent Schema Registry, schema compatibility, Kafka Connect, connectors, ACLs, and service accounts.

• Knowledge of Kafka security and authentication mechanisms, including Kerberos, SASL, TLS, API keys, OAuth, or mTLS.

• Experience with AWS or GCP cloud services, private networking, and secure connectivity.

• Hands-on experience with Terraform or other Infrastructure as Code tools and CI/CD automation.

• Experience with production cutovers, rollback planning, disaster recovery, monitoring, and incident troubleshooting.

• Strong understanding of data integrity, performance optimisation, security controls, and operational readiness.

• Ability to coordinate with application development, cloud infrastructure, security, networking, and operations teams.

Good-to-Have Skills

• Experience with migration of Confluent Platform to Confluent Cloud Enterprise clusters.

• Knowledge of Apache Flink architecture, stream-processing concepts, and Kafka–Flink integration.

• Exposure to multi-cloud or cross-region Kafka migration strategies.

• Experience with AWS PrivateLink, GCP Private Service Connect, VPC peering, VPN, or Direct Connect.

• Knowledge of ksqlDB, Tableflow, and Confluent Cloud governance capabilities.

• Experience with Prometheus, Grafana, Instana, or similar observability tools.

• Knowledge of Kubernetes, Red Hat OpenShift, GitOps, and cloud-native deployment practices.

• Experience with managed streaming services and enterprise event-driven architectures.

• Confluent Certified Developer or Administrator for Apache Kafka certification.

• AWS or GCP cloud certification and HashiCorp Terraform Associate certification.

Desired Skills

• Systems Analysis
• Complex Problem Solving
• Programming/configuration
• Critical Thinking
• Time Management