Databricks Developer

Infinite Computer Solutions - Morris Plains, NJ

Full-time

Morris Plains, NJ

Professional, Scientific, and Technical Services

About the position

The Databricks Developer role involves working with various data processing technologies and tools to develop and optimize data pipelines and analytics solutions. The position focuses on leveraging Databricks and Spark technologies to handle large datasets and implement machine learning models, ensuring efficient data processing and integration with cloud services.

Responsibilities

• Develop and optimize data pipelines using Databricks and Spark technologies.
• Implement Spark RDD, Data Frames, and SparkSQL for data processing.
• Utilize PySpark for data manipulation and analysis.
• Work with Spark Streaming for real-time data processing.
• Apply Spark ML for machine learning model development.
• Integrate data solutions with Snowflake and AWS Cloud services.
• Manage CI/CD pipelines using Jenkins and Terraform for deployment.

Requirements

• Proficiency in Spark RDD, Data Frames, and SparkSQL.
• Experience with PySpark and Spark Streaming.
• Knowledge of Spark ML for machine learning applications.
• Familiarity with Databricks platform.
• Experience with Snowflake for data warehousing.
• Understanding of AWS Cloud services and DevOps practices.
• Experience with Jenkins CI/CD pipelines and Terraform.