This Platform Architect role focuses on Azure Databricks, overseeing its design, deployment, and optimization. The position involves managing data governance, security, networking, and integration with other Azure services, ensuring robust and cost-effective solutions. It suits an experienced professional with deep expertise in Azure services and Databricks specifics.
Responsibilities
Architect and oversee the deployment of Azure Databricks workspaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints
Define and manage Databricks cluster policies, instance types, Databricks Runtime (DBR) versions, and auto-scaling configurations
Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations
Forecast and plan for future Databricks resource needs based on platform usage trends
Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances)
Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems
Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations
Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required
Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access
Advise and govern the usage of managed and external tables/volumes within Unity Catalog
Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization
Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks
Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications
Advise on credential passthrough configurations
Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access
Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments
Ensure Databricks configurations meet organizational security and compliance standards
Assist in auditing access logs and activity
Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs)
Millions of jobs, with real people getting hired every day
200,000+
New jobs added daily
7,000,000+
Verified job listings
1,000,000+
Tailored applications submitted
Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources
Ensure proper DNS resolution and routing for Databricks components and integrated services
Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines
Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps
Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed
Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies
Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals
Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization)
Contribute to and validate disaster recovery and business continuity plans for the Databricks platform
Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities
Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks
Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform
Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams
Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage
Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues
Act as a subject matter expert for all Databricks-related queries from internal teams
Participate in architectural review boards and provide technical leadership
Requirements
10+ years of experience
Deep expertise in Azure Services: Azure Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, Azure Networking (VNets, NSGs, Private Endpoints), Azure Entra ID (formerly Azure AD), Azure Monitor, Log Analytics
Strong understanding of cloud security principles, identity and access management (IAM), role-based access control (RBAC), data encryption, network security
Experience with SCIM
Solid understanding of TCP/IP, VPNs, routing, firewalls, and secure network design in Azure
Proficiency in Python, Scala, SQL, and PowerShell/Azure CLI for automation and management tasks
Experience with CI/CD pipelines for cloud resources and Databricks artifacts (e.g., Azure DevOps, GitHub Actions)
Excellent analytical and problem-solving skills, with the ability to troubleshoot complex, multi-service issues
Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences
Legally authorized to work in the United States
Nice to have:
Microsoft Certified: Azure Solutions Architect Expert
Databricks Certified Data Engineer Professional/Associate
Important Details
Contract: Contract
Work format: Remote
About the Company
Themesoft Inc is an Equal Employment Opportunity employer.
