Data Engineer

Starcom Consulting - Los Angeles, CA

Full-time

Los Angeles, CA

Professional, Scientific, and Technical Services

About the position

The Data Engineer role involves designing, building, and maintaining efficient data pipelines to process and store data from various sources. The position requires collaboration with cross-functional teams to define data requirements and deliver solutions that enhance merchandising and sales. The role also emphasizes improving data quality and contributing to the broader Data Engineering community.

Responsibilities

• Design, build, and maintain robust and efficient data pipelines that collect, process, and store data from various sources.
• Develop data models for efficient analysis and manipulation of data for merchandising optimization.
• Ensure data quality, consistency, and accuracy.
• Build scalable data pipelines using SparkSQL Scala and Airflow scheduler/executor framework.
• Collaborate with cross-functional teams to define data requirements and deliver data solutions.
• Contribute to the broader Data Engineering community to influence tooling and standards.
• Improve code and data quality by leveraging internal tools to detect and mitigate issues.

Requirements

• 5-9+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD.
• Experience with distributed processing technologies and frameworks such as Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3).
• Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective product solutions.
• Expertise with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks.
• Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse).
• Excellent written and verbal communication skills.

Nice-to-haves

• Expertise in Python and SQL (expert level).
• Intermediate knowledge of Spark and Scala.