Senior Data Engineer Python and/or Spark, Boston, MA

Blueskyclarity - Boston, MA

Full-time - Mid Level

Boston, MA

About the position

The Senior Data Engineer will play a crucial role in the Product & Technologies group, focusing on building and enhancing the data integration framework within a modern cloud-based data hub architecture. This position involves collaborating with senior engineers to design scalable solutions for diverse data formats and ensuring robust system performance through post-production support.

Responsibilities

• Build out the data integration framework capability within the data hub architecture.
• Collaborate with senior engineers to design and engineer a robust, scalable integration framework.
• Provide post-production monitoring and support for the components built.
• Develop a sound understanding of the existing architecture.
• Create additional data hub infrastructure components to enhance scalability and robustness.
• Build frameworks, tooling, and pipelines to meet partner data integration requirements.
• Provide operational support for the framework/tooling in production during early stages.
• Use agile/scrum methodology to ensure regular sprint commitments are met.
• Perform PR reviews for other engineers.
• Stay updated on new services and technologies to evolve the data hub architecture.

Requirements

• Strong background in data engineering with experience in designing and implementing data pipelines for large-scale transformations.
• Degree in Computer Science or related field.
• Solid understanding of algorithms and data structures.
• Strong understanding of relational databases.
• Proficient in ANSI SQL and procedural SQL.
• Proficient in at least one programming language: Python, Java, or Scala.
• Experience with Apache Spark for large-scale data processing.
• Familiarity with AWS services such as AWS Glue, EMR, and S3 is a strong plus.
• Experience with data lake architectures is desirable.
• Good understanding of MPP architectures and the Hadoop Big Data ecosystem.
• Experience working with unstructured data is a plus.

Nice-to-haves

• Experience with data lake type architecture.
• Familiarity with AWS services such as AWS Glue, EMR, and S3.

Benefits

• Compensation commensurate with experience
• Bonus
• Benefits
• Equal Opportunity Employer (EOE)
• Diverse and highly educated culture