About This Role
Designing, developing, and maintaining scalable data pipelines and engineering solutions is the core of this role. It involves building and optimizing data processing and analytical solutions using cloud-based technologies like Snowflake, Databricks, and AWS services, while collaborating with business and technical teams to meet data requirements.
Responsibilities
- Design, develop, and maintain scalable data pipelines and data engineering solutions
- Develop data ingestion, transformation, and processing workflows using cloud-based technologies
- Work with Snowflake and Databricks to build and optimize data processing and analytical solutions
- Develop and maintain data pipelines using AWS Glue, S3, and Lambda
- Implement data transformation, cleansing, validation, and integration processes
- Develop efficient SQL queries and optimize data processing and warehouse workloads
- Work with business and technical teams to understand data requirements and translate them into technical solutions
- Participate in solution design, development, testing, and deployment activities
- Develop reusable components and follow established data engineering and coding standards
- Troubleshoot data pipeline and application issues and implement appropriate fixes
- Participate in code reviews, technical discussions, and continuous improvement initiatives
- Support deployment and release activities across environments
- Monitor data pipelines and proactively identify issues affecting data processing or availability
- Provide support for production issues when required, including troubleshooting and root cause analysis
- Maintain technical documentation and operational procedures
Requirements
- 3-8 years of hands-on experience in data engineering and ETL/ELT development
- Strong understanding of data engineering concepts, ETL/ELT, and data pipelines
- Hands-on experience with Snowflake and/or Databricks
- Experience building data pipelines and data transformation workflows
- Strong SQL skills
- Good understanding of data modelling and data warehousing concepts
- Good experience with cloud-based data platforms and data pipelines
- Hands-on experience with one or more of the following AWS services:
- AWS Glue
- Amazon S3
- AWS Lambda
- Experience with Databricks data engineering, including:
- Data transformation and processing
- Delta Lake
- PySpark and/or Spark
- Notebook-based development
- Experience with Snowflake development, including:
