Senior Databricks Data Engineer

• Experience: 6-8 years of dedicated experience in Data Engineering, with a strong focus on modern data platform architectures.
• Databricks Expertise: Deep hands-on experience with the Databricks Lakehouse platform, Delta Lake, and Unity Catalog implementation.
• Core Languages: Exceptional proficiency in SQL (advanced querying, tuning, data modeling) and PySpark (structured streaming, DataFrame API).
• Cloud Ecosystem: Strong exposure to the AWS cloud platform and its core data/compute infrastructure services.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.

Roles & Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable, reliable, and secure batch and real-time data pipelines using PySpark and SQL within the Databricks platform.
• Architecture & Optimization: Optimize complex SQL queries and PySpark jobs for performance, cost, and memory efficiency.
• Data Governance: Implement and manage data governance, access controls, and data discovery workflows using Databricks Unity Catalog.
• Cloud Integration: Design and deploy data solutions leveraging the AWS Cloud ecosystem (e.g., S3, IAM, Lambda, EC2, CloudWatch, or Redshift).
• Client & Stakeholder Management: Serve as a primary technical point of contact for clients. Gather requirements, present technical solutions, manage expectations, and provide regular project updates.
• Mentorship & Best Practices: Promote best practices in code quality, version control (Git), CI/CD, and data testing frameworks within the engineering team.