About This Role
Designing scalable data pipelines and optimizing Databricks SQL workloads for cloud-based analytics solutions are the core responsibilities of this hybrid role, which combines data engineering and data architecture. The role focuses on ensuring robust data models that support business analytics across life and annuities insurance domains, using PySpark and Databricks platforms. The candidate will collaborate with cross-functional teams during day shifts to deliver high-quality data products that enable informed decisions.
Responsibilities
- Design and implement scalable data pipelines using PySpark to ingest, transform, and validate large volumes of structured and semi-structured data within Databricks workspaces, ensuring reliable and reusable data assets across business units.
- Develop and maintain optimized Databricks SQL queries and data models that support analytics, reporting, and self-service data consumption, focusing on performance tuning and cost-efficient resource utilization in the shared compute environment.
- Architect end-to-end data solutions by defining data lake and warehouse structures, partitioning strategies, and storage conventions so that data remains consistent, discoverable, and compliant with enterprise data governance practices.
- Collaborate closely with data analysts, business stakeholders, and application teams to translate analytical and regulatory requirements into robust technical designs that improve accuracy, timeliness, and usability of critical data sets.
- Implement rigorous data quality frameworks, including profiling, validation rules, and automated checks in PySpark and Databricks workflows to reduce data defects and ensure trustworthy information for downstream analytics consumers.
- Configure and maintain Databricks jobs, clusters, and notebooks by establishing standards for environment configuration, code packaging, and monitoring so that pipelines run predictably and are straightforward to troubleshoot.
- Optimize data processing workloads by analyzing job execution metrics, refining PySpark transformations, and reorganizing data storage patterns to enhance throughput, minimize latency, and support evolving business demand without service disruption.
- Document data architectures, pipeline flows, and dataset definitions in a clear and consumable format so technical and non-technical partners can understand data lineage, usage constraints, and the impact of changes on reporting and regulatory outcomes.
- Support secure data handling practices by applying access controls, encryption configurations, and anonymization techniques in Databricks so sensitive information remains protected while still enabling meaningful analysis and innovation.
- Coordinate with enterprise data governance and compliance teams to align data solutions with standards for quality, retention, and auditability, especially where life and annuities insurance data influences pricing, risk assessment, and customer servicing decisions.
