About the role
Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.
- Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
- Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
- Implement robust checks for Data Quality Dimensions
- Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
- Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
- Implement data quality-as-code practices using tools like Soda, dbt etc.
- Validate Spark transformations, Delta Lake tables, and streaming pipelines
- Perform large-scale data validation using PySpark and SQL
- Optimize validation logic for high-volume datasets
- Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
- Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
- Define and implement data quality SLAs and KPIs
- Build dashboards to track data health and pipeline reliability
- Proactively identify anomalies and data drifts
- Implement alerting mechanisms for data failures
- Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
- Translate business requirements into technical reporting solutions.
- Identify bottlenecks and improve report/query performance.
- Implement best practices for dashboard design and scalability.
- Establish BI standards, frameworks, and best practices.
Required Qualifications
- 6-8 years experience in enterprise data modeling and data architecture roles.
- Strong hands-on experience with Azure Data Factory & Pyspark in large-scale environments is preferred.
- Experience with Databricks (Delta Lake, Spark, Unity Catalog).
- Advanced SQL and strong understanding of pyspark scripts & using pyspark for Data validation.
- Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
- Strong understanding of data governance, data quality, metadata, and lineage.
- Excellent communication skills across business and technical audiences.
Core Technology Stack
- Analytics Platform: Databricks (Delta Lake, Unity Catalog), Microsoft PowerBI
- Languages: SQL, Spark SQL, Python / PySpark
- Governance/Data Quality: Unity Catalog/Informatica DG/DQ
#LI-KS1
Millions of jobs, with real people getting hired every day
Questions, answered
Click "Apply with JobAssist" – we tailor your resume and application to this role and submit it for your approval.
Yes. This role at Wesco was screened before publishing – we confirmed the employer before listing it.
The employer didn't disclose a salary range for this listing. JobAssist shows pay whenever it's available.
This position can be done from anywhere, with no in-office requirement.
Yes – every application is tailored from your profile and this job's requirements, and you can review and edit before it's sent.