Data Engineer II

Beacon Hill Staffing Group - Saint Paul, MN

Full-time - Mid Level

Saint Paul, MN

Administrative and Support Services

About the position

The position involves leading the design, development, and implementation of ETL pipelines using Apache NiFi and Java Spring Boot microservices. The role focuses on integrating large volumes of complex data from diverse sources while ensuring optimal performance and scalability of existing ETL processes. The candidate will collaborate with data scientists and machine learning engineers, maintain comprehensive data documentation, and establish best practices for secure data management and compliance.

Responsibilities

• Lead the design, development, and implementation of ETL pipelines using Apache NiFi and Java Spring Boot microservices.
• Integrate large volumes of complex data from diverse sources.
• Optimize existing ETL processes for performance and scalability.
• Conduct analyses for effective data ingestion strategies.
• Collaborate with data scientists and machine learning engineers for model support.
• Maintain comprehensive data documentation, including database design and performance reports.
• Establish best practices for secure data management and regulatory compliance.
• Partner with data management teams to define and implement data governance initiatives.
• Foster a collaborative team environment that promotes knowledge sharing.
• Communicate complex technical concepts to both technical and non-technical stakeholders.
• Represent the data engineering team in stakeholder meetings, addressing challenges.
• Delegate tasks to ensure timely and budget-compliant project completion.
• Mentor junior data engineers to enhance their skills and professional growth.

Requirements

• Deep expertise in Java programming, Spring Boot, and AWS big data technologies (e.g., Redshift, Athena).
• Strong understanding of data integration principles, ETL best practices, and data governance frameworks.
• Familiarity with machine learning concepts and data preparation techniques.

Nice-to-haves

• Experience with Apache NiFi.
• Knowledge of data governance frameworks.

Benefits

• Equal Opportunity Employer that values diversity.
• Encouragement for individuals with disabilities and protected veterans to apply.