Databricks Developer

Donato Technologies - Irving, TX

Full-time

Irving, TX

Professional, Scientific, and Technical Services

About the position

The Databricks Developer role focuses on developing and implementing scalable data pipelines and ETL processes using Databricks and Apache Spark. The position requires collaboration with data science and engineering teams to manage large datasets in a cloud environment, optimizing performance and resource efficiency.

Responsibilities

• Develop and implement scalable data pipelines and ETL processes using Databricks and Apache Spark.
• Integrate, transform, and manage large datasets from various sources in a cloud environment (Azure, AWS).
• Collaborate with data science and engineering teams to streamline data processing and model integration.
• Optimize and monitor Databricks clusters and jobs for performance and resource efficiency.
• Automate workflows and integrate Databricks with CI/CD pipelines using infrastructure-as-code tools like Terraform.

Requirements

• Experience with Databricks and Apache Spark for data processing.
• Proficiency in cloud environments such as Azure and AWS.
• Strong understanding of ETL processes and data pipeline development.
• Experience with CI/CD practices and infrastructure-as-code tools like Terraform.