Azure Databricks Engineer -- SINDC5840723

Job Summary

• Design, develop, and optimize scalable data engineering solutions using Azure Databricks.
• Build end-to-end ETL/ELT pipelines and Lakehouse architectures.
• Implement data governance, security, and performance optimization best practices.
• Collaborate with cross-functional teams in a consulting environment to deliver enterprise data solutions.

Responsibilities

• Design and develop ETL/ELT pipelines using Azure Databricks.
• Build batch ingestion using Auto Loader and real-time ingestion using Spark Structured Streaming.
• Develop and maintain Delta Lake-based data pipelines.
• Manage Unity Catalog for data governance, security, and access control.
• Create and maintain catalogs, schemas, tables, materialized views, functions, and volumes.
• Develop Slowly Changing Dimensions (SCD Type 1 & Type 2) for dimensional data.
• Build Change Data Capture (CDC) pipelines for incremental data processing.
• Implement Lakehouse Federation and configure foreign catalogs for external data integration.
• Optimize data storage using partitioning and Liquid Clustering.
• Ensure data quality, integrity, and consistency across data platforms.
• Participate in CI/CD implementation and follow DevOps best practices.
• Collaborate with business and technical stakeholders to deliver scalable data solutions.

Mandatory Skills

• 7–10 years of experience in Data Engineering.
• Hands-on experience with Azure Databricks and Delta Lake.
• Strong expertise in ETL/ELT pipeline development.
• Experience with Spark Structured Streaming and Auto Loader.
• Strong knowledge of Unity Catalog.
• Proficiency in SQL and data modeling.
• Strong understanding of Data Warehousing concepts.
• Experience with Microsoft Azure cloud services.
• Excellent analytical and problem-solving skills.