AWS Data Engineer

Aloden - San Francisco, CA

Full-time

San Francisco, CA

About the position

As an AWS Data Engineer, you will be responsible for designing, developing, and maintaining the data infrastructure necessary for onboarding data sources into a data lake, building robust data pipelines, and ensuring data quality and governance. Your role will leverage AWS services and data engineering best practices to drive data initiatives forward, collaborating with various stakeholders to deliver high-quality data solutions.

Responsibilities

• Onboard various data sources into the data lake, ensuring seamless integration and data consistency.
• Design, develop, and maintain scalable and efficient data pipelines using AWS services such as Lambda, Step Functions, and EMR.
• Register data sources and manage metadata to ensure data discoverability and accessibility.
• Implement data quality checks and transformations to ensure the accuracy and reliability of data.
• Comply with data governance principles and best practices to ensure data security, privacy, and compliance.
• Utilize Terraform scripting to manage and automate AWS infrastructure.
• Leverage Spark and other big data technologies to process and analyze large datasets.
• Use Airflow and Step Functions to orchestrate complex data workflows.
• Work with Snowflake, Iceberg table formats, and other data modeling tools to design and optimize data storage solutions.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.

Requirements

• Proficiency in AWS Lake Formation, Step Functions, Lambda (serverless), EC2, EMR, and EKS.
• Strong experience with Python and Terraform scripting.
• Experience with Jupyter Notebook, RDS, Snowflake, and Iceberg table formats.
• Expertise in Spark and data pipeline orchestration tools like Airflow and dbt.
• Solid understanding of data engineering principles, including ETL processes, data warehousing, and data modeling.
• Knowledge of data governance principles and best practices.
• Strong analytical and problem-solving skills with the ability to troubleshoot and resolve data-related issues.
• Excellent communication skills with the ability to collaborate effectively with cross-functional teams.

Nice-to-haves

• AWS Certified Data Engineer or Analytics - Specialty, AWS Certified Solutions Architect, or other relevant certifications.
• Previous experience in a similar role within a fast-paced, data-driven environment.