Data Engineer

Aloden - Columbus, OH

Full-time

Columbus, OH

About the position

The Data Engineer role involves designing, developing, and maintaining data pipelines primarily using PySpark and Java. This position is crucial for migrating data from legacy systems to a modern AWS-based platform, ensuring data quality and integrity while collaborating with data scientists and stakeholders.

Responsibilities

• Design, develop, and maintain data pipelines using PySpark and Java.
• Migrate data from legacy platforms to a new AWS-based platform.
• Write and optimize complex Spark transformations and SQL queries.
• Work collaboratively with data scientists and other stakeholders.
• Ensure data quality, integrity, and security.

Requirements

• Strong proficiency in Python and PySpark (70% PySpark, 30% Java focus).
• Solid understanding and experience in Java development.
• Strong SQL skills for data querying and manipulation.
• Extensive experience with AWS cloud services and its data-related offerings.
• Experience working with and building data lakes.
• Hands-on experience with data processing and transformation using PySpark.
• Excellent problem-solving and communication skills.

Nice-to-haves

• Experience with Databricks.