Power Platform Architect

About This Role

This Platform Architect role focuses on Azure Databricks, overseeing its design, deployment, and optimization. The position involves managing data governance, security, networking, and integration with other Azure services, ensuring robust and cost-effective solutions. It suits an experienced professional with deep expertise in Azure services and Databricks specifics.

Responsibilities

• Architect and oversee the deployment of Azure Databricks workspaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints
• Define and manage Databricks cluster policies, instance types, Databricks Runtime (DBR) versions, and auto-scaling configurations
• Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations
• Forecast and plan for future Databricks resource needs based on platform usage trends
• Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances)
• Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems
• Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations
• Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required
• Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access
• Advise and govern the usage of managed and external tables/volumes within Unity Catalog
• Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization
• Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks
• Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications
• Advise on credential passthrough configurations
• Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access
• Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments
• Ensure Databricks configurations meet organizational security and compliance standards
• Assist in auditing access logs and activity
• Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs)
• Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources
• Ensure proper DNS resolution and routing for Databricks components and integrated services
• Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines
• Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps
• Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed
• Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies
• Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals
• Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization)
• Contribute to and validate disaster recovery and business continuity plans for the Databricks platform
• Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities
• Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks
• Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform
• Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams
• Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage
• Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues
• Act as a subject matter expert for all Databricks-related queries from internal teams
• Participate in architectural review boards and provide technical leadership

Requirements

• 10+ years of experience
• Deep expertise in Azure Services: Azure Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, Azure Networking (VNets, NSGs, Private Endpoints), Azure Entra ID (formerly Azure AD), Azure Monitor, Log Analytics
• Databricks Specifics: Unity Catalog, Delta Lake, Spark architecture, Databricks Runtime, Workspaces, Clusters, Jobs, Notebooks, Repos, Databricks APIs
• Strong understanding of cloud security principles, identity and access management (IAM), role-based access control (RBAC), data encryption, network security
• Experience with SCIM
• Solid understanding of TCP/IP, VPNs, routing, firewalls, and secure network design in Azure
• Proficiency in Python, Scala, SQL, and PowerShell/Azure CLI for automation and management tasks
• Experience with CI/CD pipelines for cloud resources and Databricks artifacts (e.g., Azure DevOps, GitHub Actions)
• Excellent analytical and problem-solving skills, with the ability to troubleshoot complex, multi-service issues
• Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences
• Legally authorized to work in the United States

Nice to have:

• Microsoft Certified: Azure Solutions Architect Expert
• Databricks Certified Data Engineer Professional/Associate

Important Details

• Contract: Contract
• Work format: Remote

About the Company

Themesoft Inc is an Equal Employment Opportunity employer.