Databricks Developer

Donato Technologies - New York, NY

Full-time

New York, NY

Professional, Scientific, and Technical Services

About the position

The Databricks Developer role at Donato Technologies involves building and implementing data pipelines using Databricks or similar cloud databases. The position requires expertise in SQL for writing complex queries and hands-on experience with object-oriented programming in Python. The developer will also work on real-time data streams using Spark and will need to understand architectural best practices for building data lakes.

Responsibilities

• Build and implement data pipelines using Databricks or similar cloud databases.
• Write complex, highly-optimized SQL queries across large volumes of data.
• Develop real-time data streams using Spark.
• Utilize object-oriented programming in Python for data processing.
• Apply architectural best practices in building data lakes.
• Work with containerization and orchestration technologies such as Docker and Kubernetes.
• Provision AWS infrastructure using Terraform and/or Cloud Formation.

Requirements

• Experience building/implementing data pipelines using Databricks or similar cloud database.
• Expert level knowledge of SQL for complex queries.
• Hands-on experience with object-oriented programming in Python.
• Professional experience with real-time data streams using Spark.
• Knowledge of architectural best practices for data lakes.
• Experience with two or more scripting languages such as Python and Bash.
• Solid understanding of containerization and orchestration technologies.
• Understanding of infrastructure as code, specifically AWS provisioning using Terraform or Cloud Formation.