Data Engineer (Azure Databricks)

HMG America - Portland, OR

Full-time - Mid Level

Portland, OR

Professional, Scientific, and Technical Services

About the position

The Data Engineer (Azure Databricks) role at HMG America LLC involves designing and implementing scalable data engineering solutions using Azure Databricks and related technologies. The position requires strong technical expertise, leadership capabilities, and the ability to work collaboratively across teams to deliver high-quality data solutions. The role is focused on managing complex data pipelines and ensuring operational effectiveness within a SaaS environment.

Responsibilities

• Manage end to end delivery by investigating problem areas, working cross-functionally with product manager & other stakeholders
• Follow the Agile development methodology; think strategically and execute methodically
• Develop and manage capacity and growth projection forecasts of the environment within budgets
• Create and maintain optimal data pipeline architecture, assembling large, complex data sets that meet functional/non-functional business requirements
• Drive optimization, testing and tooling to improve quality of solutions
• Manage teams that build and operate high volume distributed systems in a SaaS environment
• Devise efficient processes that increase velocity and quality
• Train counterparts in data pipelining and preparation techniques
• Promote available data and analytics capabilities to business unit leaders and educate them in leveraging these capabilities
• Accountable for operational effectiveness in performance, uptime, release management of the enterprise data platform

Requirements

• 7-14 years of relevant experience
• Bachelor's and/or master's degree in computer science or equivalent experience
• Strong communication, analytical and problem-solving skills with a high attention to detail
• At least two years of experience building and leading highly complex, technical engineering teams
• Strong hands-on experience in Databricks
• Experience managing distributed teams preferred
• Expertise on Azure Cloud platform
• Good SQL knowledge
• Ability to set and lead the technical vision while balancing business drivers
• Strong experience with PySpark, Python programming
• Proficiency with APIs, containerization and orchestration is a plus
• Experience handling large and complex sets of data from various sources and databases
• Solid grasp of database engineering and design principles
• Experience with Unity Catalog
• Familiarity with CI/CD methods desired

Nice-to-haves

• Good to have Teradata Experience (not Mandatory)