About the position
The Data Engineer role involves transforming data into actionable insights that drive business decisions. The successful candidate will be responsible for the full lifecycle of data analysis, including requirements gathering, design, and implementation of ETL processes. This position requires a strong understanding of data warehousing, cloud technologies, and programming, particularly in Python and SQL, to develop robust data solutions within the Azure and AWS environments.
Responsibilities
- Design and develop ETL processes based on functional and non-functional requirements in Python/Pyspark within the Azure platform.
- Understand the full end-to-end development activities from design to go-live for ETL development and Azure platform.
- Recommend and execute improvements to existing processes.
- Document component design for developers and for broader communication.
- Adopt an Agile (SCRUM-like) software development mindset.
- Follow established processes/standards and business technology architecture for development, release management, and deployment processes.
- Execute and provide support during testing cycles and post-production deployment, engaging in peer code reviews.
- Elicit, analyze, and interpret business and data requirements to develop complete business solutions, including data models, ETL, and business rules.
- Apply automation and innovation on new and ongoing data platforms for development projects aligned to business strategies.
- Design, develop, and implement reporting platforms and complex ETL frameworks that meet business requirements.
- Deliver business or enterprise data deliverables that adhere to enterprise frameworks for various platforms/servers/applications/systems.
Requirements
- Proven working experience as a data engineer.
- Bachelor's degree or equivalent in Computer Science.
- Skilled in Python object-oriented programming.
- Skilled in AWS Compute such as EC2, Lambda, Beanstalk, or ECS.
- Skilled in AWS Database products such as Neptune, RDS, Redshift, or Aurora.
- Skilled in AWS Management and Governance suite of products such as CloudTrail, CloudWatch, or Systems Manager.
- Skilled in Amazon Web Services (AWS) offerings, development, and networking platforms.
- Skilled in SQL and Jenkins.
- Skilled in JSON and discovering patterns in large data sets using relevant software such as Oracle Data Mining or Informatica.
- Skilled in cloud technologies and cloud computing.
- Experience using software and computer systems' architectural principles to integrate enterprise applications.
- Determining causes of operating errors and taking corrective action.
- Experience in analyzing data to identify trends or relationships.
- Skilled in creating and managing databases using relevant software such as MySQL, Hadoop, or MongoDB.
- Strong programming skills including coding and debugging.
- Excellent communication skills, both written and verbal.
- Ability to frame ideas as systems and analyze inputs, outputs, and processes.
- Experience in helping organizations plan and manage change to meet strategic objectives.
- Adept at managing project plans, resources, and people to ensure successful project completion.
Nice-to-haves
- Experience with Databricks.
- Familiarity with Informatica/ETL.
