Sr. Architect Databricks -----------12+ years

Devbytes inc - Boston, MA

Full-time - Senior

Boston, MA

About the position

The Sr. Architect Databricks position is a long-term contract role based in Boston, MA, requiring a highly experienced professional with a strong background in data architecture and Databricks technologies. The role involves leading the design and implementation of scalable data architectures, overseeing Databricks Workspace administration, and providing technical guidance to junior team members. The ideal candidate will work in a hybrid environment, focusing on delivering impactful data solutions that align with business objectives.

Responsibilities

• Lead the design and implementation of scalable data architectures using Databricks and Azure technologies.
• Oversee the administration of Databricks Workspace and ensure optimal performance.
• Provide expertise in Databricks CLI and Unity Catalog for efficient data management.
• Implement and manage Kafka for real-time data streaming solutions.
• Utilize Azure Monitor to track and optimize system performance.
• Manage data storage solutions using Azure Data Lake Store.
• Ensure data security and compliance with Azure Key Vault.
• Coordinate with development teams using Azure DevOps for seamless CI/CD pipelines.
• Integrate and manage event-driven architectures using Azure Event Hub.
• Develop and maintain data processing workflows using Python and PySpark.
• Create and optimize SQL queries within Databricks SQL for data analysis.
• Automate data workflows and pipelines using Databricks Workflows.
• Collaborate with cross-functional teams to deliver high-quality data solutions.
• Provide technical guidance and mentorship to junior team members.
• Drive continuous improvement initiatives to enhance data architecture and processes.
• Ensure alignment of data solutions with business objectives and goals.
• Contribute to the company's mission by delivering impactful data solutions that drive business success.

Requirements

• Possess a minimum of 10 years of experience in data architecture and related technologies.
• Demonstrate expertise in Databricks Workspace Admin, Databricks CLI, and Databricks Unity Catalog.
• Have hands-on experience with Kafka for real-time data streaming.
• Show proficiency in using Azure Monitor, Azure Data Lake Store, and Azure Key Vault.
• Exhibit strong skills in Azure DevOps for CI/CD pipeline management.
• Be experienced in Azure Event Hub for event-driven architectures.
• Have advanced knowledge of Python and PySpark for data processing.
• Be skilled in writing and optimizing SQL queries within Databricks SQL.
• Demonstrate ability to automate workflows using Databricks Workflows.
• Possess excellent problem-solving and analytical skills.
• Show strong communication and collaboration abilities.
• Be committed to continuous learning and staying updated with industry trends.
• Have a proven track record of delivering successful data solutions.