About This Role
This is a hands-on technology leadership role responsible for architecting, developing, optimizing, and operating large-scale batch and real-time data-processing platforms at an enterprise scale. The position requires a candidate who can combine deep technical expertise in distributed data processing with effective team leadership, stakeholder management, and end-to-end delivery accountability. The successful candidate will serve as a subject-matter expert in modern data engineering, guiding both technical strategy and execution.
Responsibilities
- Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
- Resolve a variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
- Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
- Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
- Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
- Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
- Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
- Lead the architecture, design, and development of high-throughput batch and real-time streaming data pipelines using Apache Spark, Databricks, and Kafka
- Serve as the senior subject-matter expert for Spark architecture, job optimization, SQL query tuning, distributed processing, and Databricks performance engineering
- Write, review, and approve clean, modular, and highly performant Scala and Python code, enforcing best practices across the team
- Diagnose and resolve complex performance bottlenecks by analyzing Spark DAGs, execution plans, and executor metrics to address issues like data skew, shuffling, and memory management
- Apply advanced optimization techniques for Spark and SQL, including broadcast joins, predicate pushdown, partition pruning, Z-Ordering, and adaptive query execution
- Manage and optimize Databricks workspaces, including Unity Catalog, Delta Live Tables, job clusters, security, governance, and monitoring
- Implement and manage Apache Iceberg or Delta Lake as an enterprise table format, applying knowledge of features such as hidden partitioning, schema evolution, and compaction strategies
- Develop and enforce enterprise data-modelling standards and Lakehouse practices, including medallion architecture (Bronze, Silver, Gold layers)
- Lead, mentor, and develop a global team of data engineers, conducting code reviews, architecture reviews, and technical design sessions
