Advanity Technologies - Irving, TX
posted about 2 months ago
Full-time
Irving, TX
Professional, Scientific, and Technical Services
About the position
The Data Integration Engineer role focuses on developing and managing data integration processes, primarily in cloud environments. The position requires extensive experience in data engineering, particularly with AWS technologies, to automate ETL processes and manage data workflows effectively. The engineer will work on data migration projects and collaborate with data science teams to support machine learning initiatives.
Responsibilities
- Develop and automate ETL processes using Python and AWS services.
- Manage AWS data infrastructure, including services like S3, Athena, EMR, Glue, Redshift, Kinesis, and SageMaker.
- Implement data migration strategies from on-premises to cloud environments.
- Collaborate with data science teams to integrate data for machine learning applications.
- Conduct testing and validation of data workflows and ETL processes.
- Utilize SQL/Unix/Linux scripting for data manipulation and workflows.
Requirements
- 10+ years of experience as a Data Engineer.
- Strong proficiency in Python and cloud technologies, preferably AWS.
- Hands-on experience with AWS services such as S3, Athena, EMR, Glue, Redshift, Kinesis, and SageMaker.
- Experience with SQL/Unix/Linux scripting for data workflows.
- Development and testing experience with ETL technologies like Ab Initio, AWS Glue, Informatica, and Alteryx.
- Experience in data migration from on-premises to cloud environments.
- Skilled in DevOps/Data Ops practices, particularly in large-scale application testing.
Nice-to-haves
- Experience using Jenkins and Gitlab for CI/CD processes.
- Familiarity with Waterfall and Agile methodologies.
- Hands-on experience with testing storage tools like S3 or HDFS.
- Proficiency with industry-standard defect and test case management tools.
