Data Engineer (Azure Databricks)

Vdart - Portland, OR

Full-time - Mid Level

Portland, OR

1,001-5,000 employees

Professional, Scientific, and Technical Services

About the position

The Data Engineer (Azure Databricks) role is focused on implementing scalable and sustainable data engineering solutions using Azure Databricks and related technologies. The position requires strong technical expertise, leadership skills, and the ability to manage complex data pipelines while collaborating with cross-functional teams. The engineer will be responsible for optimizing data workflows, ensuring operational effectiveness, and promoting data capabilities within the organization.

Responsibilities

• Manage end to end delivery by investigating problem areas, working cross-functionally with product manager & other stakeholders
• Follow the Agile development methodology; think strategically and execute methodically
• Develop and manage capacity and growth projection forecasts of the environment within budgets
• Create and maintain optimal data pipeline architecture, assembling large, complex data sets that meet functional/non-functional business requirements
• Drive optimization, testing and tooling to improve quality of solutions
• Manage teams that build and operate high volume distributed systems in a SaaS environment
• Devise efficient processes that increase velocity and quality
• Train counterparts in data pipelining and preparation techniques
• Promote available data and analytics capabilities to business unit leaders and educate them in leveraging these capabilities
• Accountable for operational effectiveness in performance, uptime, release management of the enterprise data platform.

Requirements

• 7-15 years of relevant experience
• Bachelor's and/or master's degree in computer science or equivalent experience
• Strong communication, analytical and problem-solving skills with a high attention to detail
• At least two years of experience building and leading highly complex, technical engineering teams
• Strong hands-on experience in Databricks
• Expertise on Azure Cloud platform
• Good SQL knowledge
• Strong experience with PySpark, Python programming
• Solid grasp of database engineering and design principles
• Experience with Unity Catalog
• Familiarity with CI/CD methods

Nice-to-haves

• Experience managing distributed teams
• Comfortable working with ambiguity and multiple stakeholders
• Comfortable working cross-functionally with product management and directly with customers
• Proficiency with APIs, containerization and orchestration
• Good to have Teradata Experience (not Mandatory)

Benefits

• Comprehensive solutions through industry-leading business units
• Access to a global network and industry expertise
• Recognition as a trusted advisor for businesses
• Commitment to social responsibility and sustainability