Sr Data Eng (Snow + Databricks)

E-Solutions Group - Pittsburgh, PA

Full-time - Mid Level

Pittsburgh, PA

Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services

About the position

The Senior Data Engineer position focuses on building and leading complex technical engineering teams, specifically in the realm of data engineering solutions. The ideal candidate will have a strong hands-on experience with Databricks and will be responsible for implementing scalable and sustainable data engineering solutions using a variety of tools including Databricks, Azure, Apache Spark, and Python. This role requires the creation, maintenance, and optimization of data pipelines as workloads transition from development to production, tailored for specific use cases. In addition to technical expertise, the candidate should be comfortable navigating ambiguity and collaborating with multiple stakeholders. This includes working cross-functionally with product management and engaging directly with customers to gain a deep understanding of product and customer personas. The role demands a strong technical vision that aligns with business drivers, ensuring that the engineering solutions not only meet technical requirements but also support the overall business strategy. The position also requires expertise in the Azure Cloud platform, good SQL knowledge, and familiarity with orchestrating workloads in the cloud. The candidate should have a solid grasp of database engineering and design principles, experience with Unity Catalog, and a good exposure to Snowflake. Familiarity with CI/CD methods is desired, and while experience with Teradata is beneficial, it is not mandatory. The role is designed for someone who can handle large and complex datasets from various sources and databases, leveraging their skills in PySpark and Python programming to deliver effective data solutions.

Responsibilities

• Build and lead complex technical engineering teams.
• Implement scalable and sustainable data engineering solutions using Databricks, Azure, Apache Spark, and Python.
• Create, maintain, and optimize data pipelines as workloads transition from development to production.
• Work cross-functionally with product management and directly with customers to understand product and customer personas.
• Set and lead the technical vision while balancing business drivers.
• Manage distributed teams and handle ambiguity in project requirements.
• Utilize SQL and orchestrate workloads on the cloud effectively.
• Handle large and complex datasets from various sources and databases.

Requirements

• At least two years of experience building and leading highly complex technical engineering teams.
• Strong hands-on experience in Databricks.
• Expertise on Azure Cloud platform.
• Good SQL knowledge.
• Strong experience with PySpark and Python programming.
• Solid grasp of database engineering and design principles.
• Experience with Unity Catalog.
• Familiarity with CI/CD methods is desired.
• Good exposure to Snowflake.

Nice-to-haves

• Experience managing distributed teams preferred.
• Proficiency with APIs, containerization, and orchestration is a plus.
• Good to have Teradata experience (not mandatory).