Infrastructure Engineer

About This Role

This engineering role focuses on owning and operating the Kafka estate that underpins an event-driven integration platform. It involves building, deploying, and monitoring the infrastructure, ensuring its resilience and reliability. The position is ideal for an engineer with significant production Kafka experience and strong Linux and Python scripting skills.

Responsibilities

• Own the Kafka estate, including topic design and naming standards, partitioning and keying strategy, replication, retention and compaction policy, consumer group and offset management, quotas, and broker configuration and upgrades
• Ensure platform resilience and reliability by maintaining availability during upstream vendor SaaS platform degradation, absorbing traffic spikes, and ensuring events are replayable and auditable through a correctly sized, configured, and operated Kafka infrastructure
• Engineer the platform's delivery guarantees in practice: at-least-once and exactly-once semantics, idempotent producers, transactional writes where warranted, dead-letter topics, and self-service replay tooling for recovering from downstream failures
• Build streaming observability: consumer lag, queue depth, throughput and rebalance monitoring; end-to-end correlation-ID tracing across topics; dashboards, service level objectives, and alerting
• Plan capacity, cost, and retention for a durable log with regulatory audit retention requirements, and tune it as adoption scales across the firm
• Evaluate and integrate vendor-native streaming and change-data-capture mechanisms as the platform's ingress and egress paths mature
• Own and evolve the build and release pipeline, including automated Jenkins pipelines producing packaged application installs for multiple enterprise Linux generations, signed container images, and generated documentation, promoted through internal artifact and image distribution services
• Operate the runtime estate: containerised deployments, integration gateway hosts, and the ongoing operating-system modernisation of that estate
• Engineer the infrastructure behind the platform's resilience and continuity: health detection, circuit breaking, and the reconciliation and monitoring that prove the platform recovers cleanly when the upstream vendor platform is degraded or unavailable
• Automate environment provisioning and configuration, and keep the configuration and asset inventory a genuine single source of truth
• Partner with the squad's Python developers on deployability, performance, and cost; participate in change management and production support

Requirements

• Deep, hands-on production Kafka experience of at least 4+ years, including topic and partition design, consumer group mechanics, retention and compaction, replication, performance tuning, and troubleshooting lag, rebalances, and delivery failures under load