Beacon Hill Staffing Group - Frankfort, KY
posted 2 months ago
Full-time - Mid Level
Frankfort, KY
Administrative and Support Services
About the position
The Data Engineer II role at Beacon Hill Staffing Group involves leading the design, development, and implementation of ETL pipelines, primarily using Apache NiFi and Java Spring Boot microservices. This position requires expertise in integrating large volumes of complex data from various sources and optimizing ETL processes for performance and scalability. The role also emphasizes collaboration with data scientists and machine learning engineers, as well as mentoring junior engineers.
Responsibilities
- Lead the design, development, and implementation of ETL pipelines using Apache NiFi and Java Spring Boot microservices.
- Integrate large volumes of complex data from diverse sources.
- Optimize existing ETL processes for performance and scalability.
- Conduct analyses for effective data ingestion strategies.
- Collaborate with data scientists and machine learning engineers for model support.
- Maintain comprehensive data documentation, including database design and performance reports.
- Establish best practices for secure data management and regulatory compliance.
- Partner with data management teams to define and implement data governance initiatives.
- Foster a collaborative team environment that promotes knowledge sharing.
- Communicate complex technical concepts to both technical and non-technical stakeholders.
- Represent the data engineering team in stakeholder meetings, addressing challenges.
- Delegate tasks to ensure timely and budget-compliant project completion.
- Mentor junior data engineers to enhance their skills and professional growth.
Requirements
- Deep expertise in Java programming, Spring Boot, and AWS big data technologies (e.g., Redshift, Athena).
- Strong understanding of data integration principles, ETL best practices, and data governance frameworks.
- Familiarity with machine learning concepts and data preparation techniques.
