Development Data Engineer

About the position

This position will be located within the Development and IT space and work closely with computer scientists, IT and data scientists to build, deploy and optimize data pipelines and integrate data infrastructure for enabling analytics, reporting, and machine learning workloads at scale.

Responsibilities

• Build, test, and validate robust production-grade data pipelines that can ingest, aggregate, and transform large datasets according to the specifications of the internal teams who will be consuming the data.
• Configure connections to source data systems and validates schema definitions with the teams responsible for the source data.
• Monitor data pipelines and troubleshoots issues as they arise.
• Monitor data lake environment for performance and data integrity.
• Collaborate with IT and database teams to maintain the overall data ecosystem.
• Assist data science, business intelligence, and other teams in using the data provided by the data pipelines.
• Serve as on-call for production issues related to data pipelines and other data infrastructure maintained by the data engineering team.

Requirements

• BS degree in Computer Science or related field
• 3+ years of data engineering work experience
• Experience coding in Java or Scala and build tools such as Maven, Gradle, and SBT
• Experience with SQL databases
• Experience working with HDFS or S3 storage environments
• Experience with Apache Spark or Databricks and reading and writing Parquet, Avro and JSON
• Experience working in a Unix or Linux environment, including writing shell scripts
• Experience with ETL and ELT processes in data pipelines
• Experience with Docker and Kubernetes highly preferred
• Experience with workflow orchestration tools like Apache Airflow, Control-M, or Arrow highly preferred
• Experience with Apache Kafka or Confluent is preferred

Nice-to-haves

• Experience coding in Python
• Experience with NoSQL solutions is helpful