AWS ETL Developer

Government Systems Technologies - Houston, TX

Full-time - Mid Level

Houston, TX

Professional, Scientific, and Technical Services

About the position

The AWS ETL Developer role focuses on designing, developing, and maintaining scalable ETL workflows using AWS technologies. The position requires expertise in data transformation and integration, ensuring data quality, and optimizing performance across large datasets. Collaboration with various stakeholders and maintaining technical documentation are also key aspects of this role.

Responsibilities

• Design, develop, and implement scalable ETL workflows using PySpark, Python, and AWS Glue, Databricks.
• Extract, transform, and load data from various sources to AWS S3 and Redshift.
• Identify and resolve performance bottlenecks in ETL processes, ensuring optimal performance across large datasets.
• Debug PySpark programs/Job and reverse engineer the code.
• Implement automation scripts using AWS Lambda and Step Functions to schedule and monitor data pipelines.
• Ensure data integrity and quality across all stages of the ETL pipeline.
• Work closely with data architects, analysts, and stakeholders to understand requirements and provide clear communication throughout the project lifecycle.
• Create and maintain technical documentation, including data mapping, workflow designs, and ETL processes.
• Knowledge of CI/CD pipelines and best practices in deployment automation.

Requirements

• Proficiency in PySpark and Python for ETL development.
• Experience with AWS Glue and Databricks.
• Strong understanding of data transformation and integration processes.
• Ability to identify and resolve performance issues in ETL workflows.
• Experience in debugging and reverse engineering PySpark code.
• Familiarity with AWS Lambda and Step Functions for automation.
• Knowledge of data quality assurance practices.
• Strong communication skills for collaboration with stakeholders.
• Experience in creating technical documentation.

Nice-to-haves

• Experience with AWS Redshift and S3.
• Familiarity with CI/CD practices and tools.
• Knowledge of data warehousing concepts.

Benefits

• Competitive salary based on experience.
• Opportunity to work with cutting-edge technologies in AWS.
• Flexible working hours.