Databricks Architect

Vilwaa Solutions Pvt Ltd - San Diego, CA

Full-time - Senior

San Diego, CA

About the position

The AWS Databricks Architect will lead and manage a team of engineers, providing guidance and mentorship while fostering a collaborative work environment. This role involves establishing and maintaining positive relationships with client stakeholders, consulting on project status and technical issues, and providing post-implementation support. The architect will facilitate communication between onshore and offshore teams to ensure successful project execution and timely delivery of products and deliverables.

Responsibilities

• Lead and manage a team of engineers and senior engineers.
• Establish and maintain positive relationships with client stakeholders.
• Consult with clients on project status, proposals, and technical issues.
• Provide post-implementation support to clients.
• Facilitate communication and collaboration between onshore and offshore teams.
• Ensure successful design, development, and execution of projects on time.
• Develop proof of concepts (POC) and perform capacity sizing and tool evaluation.

Requirements

• 12+ years of experience in data engineering and architecture.
• Excellent written and verbal communication skills.
• Ability to run ETL jobs in Databricks.
• Experience with Unity Catalog, Delta, MongoDB, and S3-based parquet files.
• Knowledge of Delta format and its pros and cons.
• Experience running Databricks with AWS Glue Catalog.
• Monitoring and operations experience with Databricks jobs.
• Best practices for setting up nested ETL jobs.
• Experience scheduling jobs in Databricks and passing runtime parameters.

Nice-to-haves

• Experience in a client-facing role.