Python Big Data Engineer

Aloden - Wilmington, DE

Full-time

Wilmington, DE

About the position

The Python Big Data Engineer role involves designing, developing, and maintaining scalable data pipelines in a cloud environment, specifically AWS. The position requires expertise in cloud data solutions, real-time data streaming, and data lake architecture, with a strong emphasis on utilizing Python and SQL for data processing and automation tasks.

Responsibilities

• Design, develop, and maintain scalable and efficient data pipelines in the AWS cloud environment.
• Migrate existing data solutions to AWS or build new ones from scratch.
• Leverage Databricks (or similar) to construct and manage data pipelines.
• Write and optimize complex SQL queries for data extraction, transformation, and loading.
• Utilize Python for data processing and automation tasks.
• Build and maintain real-time data streaming solutions using Spark.
• Apply architectural best practices to the design and implementation of data lakes.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions.

Requirements

• 5+ years of designing and implementing data pipelines in a cloud environment.
• 2+ years of experience migrating/developing data solutions specifically in AWS.
• 1+ year of experience building and implementing data pipelines using Databricks or a similar cloud database platform.
• Expert-level SQL skills for writing complex, optimized queries across large datasets.
• Hands-on experience with object-oriented programming in Python.
• Professional experience building real-time data streams using Spark.
• Knowledge or experience in architectural best practices for building data lakes.