ETL Data Engineer

Avance Consulting - Dallas, TX

Full-time - Mid Level

Dallas, TX

About the position

The ETL Data Engineer will play a crucial role in ensuring the quality and reliability of data through the design, implementation, and execution of comprehensive testing. This position requires a strong background in ETL processes and data integration, with a focus on utilizing big data technologies and testing frameworks to validate data against business rules.

Responsibilities

• Design, implement, and execute comprehensive testing for data quality and reliability.
• Build a comprehensive code base and business rules for testing and validation of data based on User Stories.
• Utilize ETL best practices and data quality principles in data integration processes.
• Collaborate with teams using Agile/Scrum methodologies to ensure project success.
• Work with CI/CD pipelines to streamline data engineering processes.

Requirements

• 6 to 8 years of experience as a development data engineer or similar role with an ETL background.
• Strong experience in Pyspark.
• Experience with testing frameworks.
• Knowledge of data integration/ETL best practices and data quality principles.
• Familiarity with continuous integration and continuous deployment (CI/CD) pipelines.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills.
• Experience with big data technologies such as Hadoop, Spark, and Hive.