Cloud Data Engineer-Minneapolis, MN (onsite)

Avtech Solutions - Minneapolis, MN

Full-time - Mid Level

Minneapolis, MN

Professional, Scientific, and Technical Services

About the position

The Cloud Data Engineer role involves designing and implementing data solutions on Azure, focusing on large-scale data processing and integration. The position requires expertise in various Azure services and tools, with a strong emphasis on collaboration within cross-functional teams to deliver effective data-driven solutions.

Responsibilities

• Design and implement data processing solutions using Python and Apache Spark.
• Manage ETL pipelines using Azure Data Factory.
• Build and optimize data warehousing solutions on Azure.
• Collaborate with data scientists and analysts to deliver end-to-end data solutions.
• Lead and mentor junior developers in best practices and technical skills.
• Integrate APIs for data access and processing.
• Utilize Azure Synapse Analytics for data analysis and reporting.

Requirements

• Expertise in Python and Apache Spark for large-scale data processing.
• Strong experience in Azure Synapse Analytics, including SQL Pools and Spark Pools.
• Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.
• Knowledge of Data Warehousing principles with hands-on experience on Azure.
• Experience with SQL, including complex queries, optimization, and performance tuning.
• Familiarity with CI/CD tools like Azure DevOps.
• Experience in Java for API integration and microservices architecture.
• Hands-on knowledge of Kubernetes for containerized data processing environments.
• Proficiency in Azure Data Lake Storage (ADLS) Gen2 for data storage and management.
• Experience working with APIs (REST, SOAP) and building API-based data integrations.
• Experience working in an Agile environment, using Scrum or Kanban.

Nice-to-haves

• Azure certifications in data engineering or cloud architecture.
• Experience deploying AI/ML models on cloud platforms.
• Familiarity with Data Governance best practices.