Lead Azure Data Engineer

$166,400 - $176,800/Yr

Insight Global - Charlotte, NC

Full-time - Senior

Charlotte, NC

Administrative and Support Services

About the position

The Lead Azure Data Engineer will be responsible for designing, developing, and implementing data solutions within Azure to enable actionable insights from complex datasets. This role involves leading a team of data engineers, collaborating with cross-functional teams, and enhancing the organization's data infrastructure and analytics capabilities.

Responsibilities

• Spearhead the design, development, and implementation of data solutions within Azure.
• Lead a team of data engineers and mentor junior engineers.
• Build Azure enterprise systems using Databricks and Snowflake.
• Leverage SQL, Python, Spark, and PySpark to construct robust data processing pipelines.
• Apply advanced Data Engineering principles to design and implement data loading and aggregation frameworks.
• Implement CI/CD pipelines for automating build, test, and deployment processes.
• Gather and process raw, structured, semi-structured, and unstructured data using batch and real-time processing frameworks.
• Optimize data solutions in enterprise data warehouses and big data repositories, focusing on cloud migration.
• Drive new capabilities to meet the needs of product, engineering, and business teams.
• Implement data modeling techniques to design and optimize data schemas.
• Enhance performance, reliability, and scalability of data infrastructure through continuous improvement initiatives.
• Collaborate with data scientists, analysts, and stakeholders to translate business requirements into technical solutions.
• Implement best practices for data governance, security, and compliance.

Requirements

• 10+ years of experience in a data engineering role with expertise in Azure.
• 2+ years of experience as a lead, developing and mentoring other engineers.
• Advanced proficiency in SQL, including window functions.
• Proficiency in Python, Spark, and PySpark programming languages.
• Strong experience with the Azure cloud platform.
• Extensive experience with Databricks and Azure Data Factory.
• Experience in implementing CI/CD pipelines for automation.
• Hands-on experience with big data technologies such as Hadoop, Spark, and Kafka.

Nice-to-haves

• Experience in the retail industry.
• Relevant certifications (e.g., Azure, Databricks, Snowflake).
• Experience with Snowflake and/or Microsoft Fabric.

Benefits

• Medical insurance
• Dental insurance
• Vision insurance
• Health Savings Account (HSA) options
• Flexible Spending Account (FSA) options
• Dependent Care Flexible Spending Account (DCFSA) options
• 401k retirement account access with employer matching
• Paid sick leave
• Paid time off as provided by applicable law