Data Engineer, Data Platforms

Global Partners - Waltham, MA

Full-time - Mid Level

Waltham, MA

Petroleum and Coal Products Manufacturing

About the position

The Data Engineer, Data Platforms is a crucial role within the Data Team, responsible for managing and pioneering data platforms that support the company's data and analytics engineering efforts. This position involves owning advanced big data platforms that serve as the backbone for data-centric innovations across Global Partners. The role requires expertise in various technologies and a commitment to driving data excellence through strategic guidance and best practices.

Responsibilities

• Architect and implement scalable, cloud-native data platforms for data engineering initiatives.
• Design and develop CI/CD pipelines for data infrastructure and applications using tools like Jenkins and GitLab CI.
• Implement Infrastructure as Code (IaC) practices using Terraform or CloudFormation.
• Develop and maintain data orchestration workflows using tools like Apache Airflow and Dagster.
• Collaborate with security teams to manage IAM policies and access controls across data platforms.
• Design and implement data cataloging and metadata management solutions.
• Optimize data storage and processing systems, including data lakes and warehouses like Snowflake and BigQuery.
• Develop automated tools and self-service platforms for data scientists and analysts.
• Implement observability and monitoring solutions for data pipelines using tools like Prometheus and Grafana.
• Lead the adoption of DataOps practices to streamline the data lifecycle.
• Design and implement data quality frameworks and automated testing processes.
• Continuously evaluate and integrate emerging technologies in big data and cloud computing.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field, or equivalent experience in Data Engineering, DataOps, MLOps, or Software Engineering with a minimum of three years of experience.
• Strong proficiency in designing and implementing scalable, cloud-native data platforms using Infrastructure as Code (e.g., Terraform, Docker, Kubernetes).
• Advanced programming skills in Python and strong SQL proficiency with experience in cloud data warehouses (e.g., Snowflake, BigQuery).
• Proven track record in designing and implementing CI/CD pipelines for data infrastructure using tools like Jenkins and GitLab CI.
• In-depth knowledge of big data technologies (e.g., Apache Spark, Kafka) and data orchestration tools (e.g., Apache Airflow, Dagster).
• Strong background in data security, governance, and metadata management, including IAM/RBAC policies and encryption.
• Proficiency in implementing monitoring, logging, and alerting solutions for data infrastructure (e.g., Prometheus, Grafana).
• Ability to design and develop automated tools and self-service platforms for data environment management.
• Experience in optimizing data storage and processing systems for cost-effectiveness and performance.
• Exceptional team player with strong communication skills and a willingness to mentor.

Nice-to-haves

• Familiarity with serverless architectures.
• Experience with data transformation frameworks like dbt and ETL/ELT processes in cloud environments.
• Familiarity with MLOps and integrating ML models into production.

Benefits

• Competitive salaries and opportunities for growth.
• Health, Dental, Vision, and Life Insurance.
• 401k with a matching component.
• Tuition reimbursement after 6 months of service.
• Paid volunteer time-off for community support.