About the position
We are seeking a passionate and certified Data Engineer who will transform data into actionable insights to drive business decisions. The role involves full lifecycle analysis, including requirements gathering, design, and development of reporting capabilities, while ensuring performance and quality control.
Responsibilities
- Design and develop ETL processes based on functional and non-functional requirements in Python/Pyspark within the Azure platform.
- Understand the full end-to-end development activities from design to go live for ETL development and Azure platform.
- Recommend and execute improvements to existing processes.
- Document component design for developers and for broader communication.
- Adopt an Agile (SCRUM-like) software development mindset.
- Follow established processes/standards and business technology architecture for development, release management, and deployment processes.
- Execute and provide support during testing cycles and post-production deployment, engaging in peer code reviews.
- Elicit, analyze, and interpret business and data requirements to develop complete business solutions, including data models, ETL, and business rules.
- Apply automation and innovation on new and ongoing data platforms for development projects aligned to business strategies.
- Design, develop, and implement reporting platforms and complex ETL frameworks that meet business requirements.
- Deliver business or enterprise data deliverables that adhere to enterprise frameworks for various platforms.
Requirements
- Proven working experience as a data engineer.
- Bachelor's degree or equivalent in Computer Science.
- Skilled in Python object-oriented programming.
- Skilled in AWS Compute such as EC2, Lambda, Beanstalk, or ECS.
- Skilled in AWS Database products such as Neptune, RDS, Redshift, or Aurora.
- Skilled in AWS Management and Governance suite of products such as CloudTrail, CloudWatch, or Systems Manager.
- Skilled in Amazon Web Services (AWS) offerings, development, and networking platforms.
- Skilled in SQL and Jenkins.
- Skilled in JSON and discovering patterns in large data sets using relevant software such as Oracle Data Mining or Informatica.
- Skilled in cloud technologies and cloud computing.
- Experience using software and computer systems' architectural principles to integrate enterprise applications.
- Ability to determine causes of operating errors and take corrective action.
- Experience analyzing data to identify trends or relationships.
- Skilled in creating and managing databases using software such as MySQL, Hadoop, or MongoDB.
- Strong programming skills including coding and debugging.
- Excellent communication skills, both written and verbal.
- Ability to frame ideas as systems and analyze inputs, outputs, and processes.
- Experience in helping organizations plan and manage change to meet strategic objectives.
- Adept at managing project plans, resources, and people.
Nice-to-haves
- Experience with Informatica/ETL.
