Capgemini - Chicago, IL
posted 3 months ago
The Azure Data Engineer will be responsible for developing, optimizing, and maintaining data pipelines and architectures in the Microsoft Azure environment. This role involves working closely with data scientists, analysts, and other stakeholders to ensure that the data infrastructure is robust, scalable, and supports the organization’s data-driven objectives. The ideal candidate will have a strong background in data engineering and a passion for working with cloud technologies, particularly within the Azure ecosystem. In this position, you will design, develop, and maintain data pipelines to ingest, process, and store data from various sources using Azure Data Factory, Azure Databricks, and other Azure services. You will also create and optimize data models for both transactional and analytical systems using Azure SQL Database, Azure Synapse Analytics, and Azure Cosmos DB. Implementing and managing data integration solutions to ensure data consistency and integrity across systems will be a key part of your responsibilities, leveraging tools such as Azure Data Factory and Azure Logic Apps. Additionally, you will develop and maintain data warehouses and data lakes to support large-scale data analytics and reporting using Azure Synapse Analytics and Azure Data Lake Storage. Performance tuning will be essential, as you will optimize data systems for performance and cost-efficiency, including monitoring and troubleshooting issues related to data pipelines and storage. Ensuring that data processing and storage comply with organizational policies and relevant regulations will also be critical, as you implement security best practices using Azure Security Center, Azure Key Vault, and Azure Active Directory. Collaboration is key in this role, as you will work closely with data scientists, data analysts, and business stakeholders to understand data requirements and provide insights and solutions to meet their needs. You will also automate repetitive tasks and improve operational efficiency using Azure DevOps, CI/CD pipelines, and infrastructure-as-code (IaC) tools like Terraform or ARM templates. Finally, you will create and maintain comprehensive documentation of the data architecture, processes, and configurations to support ongoing operations and future developments.