Lead Azure Data Engineer

Envision Technology Solutions - Plano, TX

Full-time - Mid Level

Plano, TX

About the position

The Lead Azure Data Engineer is responsible for designing, building, and maintaining scalable data pipelines and storage solutions using Azure technologies. This role focuses on implementing ETL processes, managing data integration, and ensuring data security and compliance. The engineer will collaborate with various stakeholders to deliver high-quality data solutions and optimize data processing for performance and cost-efficiency.

Responsibilities

• Design, build, and maintain scalable data pipelines using Azure technologies.
• Implement ETL processes to collect, clean, and transform raw data from various sources.
• Develop and optimize real-time and batch data processing solutions.
• Design and manage data storage solutions using Azure data services.
• Optimize the performance, scalability, and reliability of data storage systems.
• Integrate data from different cloud and on-premise sources for business analytics.
• Work with REST APIs and web services to bring in external data sources.
• Create and manage enterprise-level data warehouses using Azure Synapse Analytics.
• Build and manage OLAP cubes and reporting systems for analytics.
• Implement data governance policies and ensure data privacy and security compliance.
• Manage access control, encryption, and data masking for sensitive data.
• Collaborate with business analysts and data scientists to understand data requirements.
• Provide support for ad-hoc data requests and troubleshoot issues.
• Monitor data pipelines and infrastructure for performance and reliability.
• Troubleshoot and resolve issues related to data processing and integration.
• Optimize data solutions for cost-efficiency and performance.
• Document data engineering workflows and stay updated with best practices.

Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• 6+ years of experience as a Data Engineer or similar role with a focus on Azure technologies.
• Proven experience with Azure Data Services (Data Factory, Data Lake, Synapse Analytics, SQL Database, Cosmos DB).
• Experience with ETL tools and data integration techniques.
• Solid understanding of data warehousing concepts and OLAP systems.
• Familiarity with programming languages like Python, SQL, or Spark.
• Proficiency in SQL and database technologies.
• Experience with data pipeline orchestration tools like Azure Data Factory or Apache Airflow.
• Knowledge of big data tools such as Azure Databricks, Apache Spark, Hadoop, and Kafka.
• Familiarity with cloud security best practices.

Nice-to-haves

• Microsoft Certified: Azure Data Engineer Associate.
• Microsoft Certified: Azure Solutions Architect Expert.
• Familiarity with machine learning pipelines and Azure Machine Learning.
• Experience working in Agile or DevOps environments.
• Exposure to data governance and master data management concepts.