DATA Engineer with Spark & Scala

E-Solutions Group - Bloomfield, CT

Full-time

Bloomfield, CT

Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services

About the position

The Data Engineer with Spark & Scala role is focused on developing scalable and innovative data solutions using technologies such as Spark, Scala, and AWS Cloud. The position requires a strong understanding of data infrastructure, data modeling, and the ability to build end-to-end ETL pipelines. The role is based in Bloomfield, CT and requires on-site presence from day one.

Responsibilities

• Develop production code as a Data or Software Engineer with 8+ years of experience.
• Deliver solid, scalable, and innovative data solutions.
• Understand and implement data infrastructure and related technologies.
• Model data and build end-to-end ETL pipelines.
• Utilize Apache Spark and related APIs (DataFrame/Dataset).
• Work with various data storage formats such as parquet, avro, and json.
• Leverage advanced SQL for data manipulation and querying.
• Utilize the Databricks platform for data engineering tasks.
• Implement CI/CD processes using Terraform, GitLab CI, and Docker.
• Design streaming and event-based systems.

Requirements

• 8+ years of experience as a Data or Software Engineer.
• Proficiency in Spark and Scala.
• Experience with Databricks and AWS Cloud technologies (S3, DynamoDB, RDS, Lambda).
• Strong understanding of data modeling and data warehousing.
• Experience in building end-to-end ETL pipelines.
• Advanced SQL experience.
• Knowledge of different data storage formats (parquet, avro, json).
• Experience with scripting languages (shell, YAML, etc).
• CI/CD experience with tools like Terraform and GitLab CI.
• Familiarity with search technologies such as Open Search.