Engineering Manager - PySpark

$192,000 - $260,000/Yr

Databricks - Mountain View, CA

Full-time - Senior

Mountain View, CA

5,001-10,000 employees

Professional, Scientific, and Technical Services

About the position

The Engineering Team Lead at Databricks will spearhead PySpark development initiatives, focusing on both open source and Databricks-specific components. The role aims to solidify Apache Spark's position within the Python community and enhance user experience through Generative AI. The lead will oversee a talented engineering team, drive product development, and promote the adoption of Apache Spark and the Databricks Data Intelligence Platform among Python users.

Responsibilities

• Lead a talented engineering team in PySpark development.
• Promote the adoption of Apache Spark and the Databricks Data Intelligence Platform among Python users.
• Oversee recruitment of top-tier talent and foster a synergistic team structure.
• Implement processes to execute product vision, strategy, and roadmap.
• Drive the integration of Generative AI into Apache Spark to improve user experience.
• Ensure consistent delivery against milestones and alignment with product leadership.
• Manage technical debt and long-term architecture decisions.

Requirements

• 5+ years experience in a related system, including Apache Spark and database internals.
• Practical experience with LLM/generative AI models.
• Experience ensuring high quality and reliable infrastructure services.
• Responsible for testing, quality, and SLAs of a product.
• Previous experience building and leading teams in complex technical domains.
• Ability to attract, hire, and coach engineers meeting Databricks standards.
• Experience managing distributed teams preferred.
• Comfort working cross-functionally with product management and customers.

Nice-to-haves

• Experience with distributed data systems or database internals.
• Ability to grow leaders and help struggling team members.

Benefits

• Comprehensive benefits package tailored to employee needs.
• Eligibility for annual performance bonus and equity.