Azure Data Engineer

$124,800 - $145,600/Yr

Outdefine - San Francisco, CA

Full-time - Mid Level

Onsite - San Francisco, CA

11-50 employees

About the position

We are seeking a skilled Azure Databricks Engineer to join our dynamic team. In this role, you will be responsible for designing, developing, and implementing big data solutions using Azure Databricks. You will collaborate with data scientists, analysts, and other stakeholders to ensure efficient data processing, transformation, and analysis.

Responsibilities

• Design and implement data pipelines and workflows using Azure Databricks.
• Develop and optimize Spark applications for data processing and analysis.
• Integrate Azure Databricks with other Azure services such as Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics.
• Perform data ingestion, cleansing, and transformation tasks.
• Monitor and troubleshoot performance issues in data pipelines.
• Collaborate with data scientists and analysts to understand data requirements and deliver appropriate solutions.
• Ensure data security and compliance with industry standards and best practices.
• Document technical solutions and processes.

Requirements

• Bachelor's degree in Computer Science, Engineering, or related field.
• Proven experience with Azure Databricks and Apache Spark.
• Strong knowledge of Azure cloud services and architecture.
• Proficiency in programming languages such as Python, Scala, or SQL.
• Experience with big data technologies and data warehousing concepts.
• Familiarity with data modeling and ETL processes.
• Excellent problem-solving and analytical skills.
• Strong communication and teamwork abilities.

Nice-to-haves

• Azure certifications (e.g., Azure Data Engineer Associate).
• Experience with CI/CD tools and practices.
• Knowledge of machine learning and data science concepts.
• Familiarity with other big data platforms such as Hadoop or Kafka.

Benefits

• Fee-free experience
• Set your own rate
• Keep all your pay