Azure with Databricks Engineer

Iquest Solutions - Philadelphia, PA

Full-time

Philadelphia, PA

Professional, Scientific, and Technical Services

About the position

The Azure with Databricks Engineer is responsible for designing, implementing, and managing Azure-based solutions that ensure scalability, reliability, and security. This role involves working with various Azure services to build robust data processing and analytics solutions, collaborating with data scientists and analysts, and optimizing data workflows.

Responsibilities

• Design, implement, and manage Azure-based solutions, ensuring scalability, reliability, and security.
• Work with Azure services such as Azure Databricks, Azure Data Lake Storage, Azure SQL Database, and others to build robust data processing and analytics solutions.
• Develop and maintain data engineering and data processing workflows using Databricks.
• Collaborate with data scientists and analysts to implement machine learning models and analytical solutions on the Databricks platform.
• Design, develop, and implement ETL processes to move and transform data from various sources to Azure Databricks.
• Optimize and troubleshoot data pipelines to ensure efficient and reliable data processing.
• Implement and adhere to security best practices for Azure and Databricks, ensuring data protection and compliance with industry standards and regulations.
• Monitor and address security vulnerabilities and potential risks within the Azure and Databricks environments.
• Collaborate with cross-functional teams, including data scientists, analysts, and other engineers, to understand project requirements and deliver effective solutions.
• Communicate technical concepts and solutions to both technical and non-technical stakeholders.
• Stay up-to-date with the latest Azure and Databricks features and updates.
• Continuously identify opportunities for process improvement, automation, and optimization of data workflows.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience working with Azure cloud services, particularly Azure Databricks.
• Strong proficiency in programming languages such as Python, Scala, or SQL.
• Experience with data engineering, ETL processes, and data integration.
• Familiarity with machine learning concepts and frameworks is a plus.
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills.
• Relevant certifications in Azure and Databricks are highly desirable.