Azure Data Engineer (Lead)

Infogain - Plano, TX

Full-time - Mid Level

Plano, TX

Professional, Scientific, and Technical Services

About the position

The Azure Data Engineer (Lead) role involves designing, developing, and maintaining ETL pipelines using Databricks, PySpark, and Azure Data Factory (ADF). The position focuses on extracting, transforming, and loading data from various sources, optimizing workflows for performance and scalability, and working with large datasets.

Responsibilities

• Design, develop, and maintain ETL pipelines using Databricks, PySpark, and ADF.
• Extract, transform, and load data from various sources.
• Optimize and fine-tune existing ETL workflows for performance and scalability.
• Work with Delta tables, deduplication, and merging with terabyte datasets.
• Utilize SQL for complex joins, subqueries, functions, and procedures.

Requirements

• Proficient in PySpark and programming skills.
• Experience with Delta tables and handling large datasets.
• 2 to 3 years of experience in Azure Data Factory (ADF).
• Strong SQL skills with experience in complex queries.