Product Support Lead Azure Databricks Data Warehouse Operations

Armo Consultants - Jersey City, NJ

Full-time

Jersey City, NJ

Professional, Scientific, and Technical Services

About the position

The Product Support Lead for Azure Databricks Data Warehouse Operations is responsible for maintaining, monitoring, and troubleshooting data pipelines and batch processes within a complex architecture. This role ensures the smooth operation of critical data workflows, addresses real-time issues, and collaborates with cross-functional teams to enhance the data infrastructure.

Responsibilities

• Monitor and ensure the health and stability of data pipelines and batch jobs running across Azure, Databricks, and TWS.
• Proactively identify, investigate, and resolve data pipeline failures or performance bottlenecks.
• Lead the triage and troubleshooting of production issues, working closely with engineering and operations teams.
• Collaborate with data engineers to optimize the performance and scalability of data workflows.
• Ensure adherence to SLAs and report on pipeline performance metrics.
• Maintain and update operational runbooks, documenting key workflows and troubleshooting steps.
• Manage the lifecycle of TWS job schedules, ensuring seamless batch processing and on-time execution.
• Provide leadership and mentorship to junior support team members.
• Develop and implement continuous improvement strategies for data operations.

Requirements

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 15+ years of experience in a data operations or product support role, preferably in data warehouses or cloud-based environments.
• Strong experience with Azure cloud services, including data storage (e.g., Azure Data Lake, Blob Storage) and compute (e.g., Azure Databricks).
• Hands-on experience with Databricks for data engineering, ETL processing, and pipeline orchestration.
• Deep understanding of TWS (Tivoli Workload Scheduler) for batch processing and scheduling.
• Strong troubleshooting skills, with the ability to quickly diagnose and resolve production issues.
• Knowledge of database technologies (SQL, NoSQL) and data integration tools.
• Excellent communication skills, with the ability to work collaboratively across technical and business teams.
• Strong organizational skills and the ability to manage multiple priorities in a fast-paced environment.

Nice-to-haves

• Experience with Python or Scala for data processing and scripting.
• Familiarity with CI/CD pipelines and DevOps practices in data environments.
• Knowledge of ITIL processes and best practices for incident and problem management.
• Experience working in an Agile/Scrum environment.