Lead Data Engineer

$130,000 - $143,712/Yr

The University of Texas System

Full-time - Mid Level

Remote

Educational Services

About the position

The Lead Data Engineer for the UT Data Hub plays a crucial role in enhancing university outcomes and advancing the mission of the University of Texas at Austin by improving the usability and value of institutional data. This position involves leading a team of data engineers to create complex data pipelines within the university's cloud data ecosystem, supporting both academic and administrative needs. The role emphasizes collaboration, creativity, and technical leadership to solve complex problems and foster a culture of learning and professional growth among team members.

Responsibilities

• Design, plan, develop and deploy production quality enterprise level data pipelines using Cloud technologies.
• Conduct and participate in code reviews to ensure high-quality code.
• Contribute to and ensure implementation of industry best practices for data engineering and SDLC.
• Identify and address bottlenecks in data pipelines, optimizing for performance and scalability.
• Develop systems to ensure data quality, consistency, cost optimization, and security compliance.
• Lead technical knowledge share sessions discussing current industry trends in Data Engineering.
• Define project milestones, timelines, and deliverables for on-time delivery.
• Collaborate with internal and external stakeholders, including Data or System Architects and business users.
• Manage, prioritize, and track team and project work while balancing urgent requests.
• Communicate and manage risks, roadblocks, and resource allocations for timely project delivery.
• Supervise a team of data engineers, providing guidance and mentorship.
• Conduct regular performance reviews and set development goals for team members.
• Promote collaboration within the team and across departments for smooth project execution.
• Provide regular updates on project status and technical challenges to both technical and non-technical audiences.
• Ensure all project deliverables are documented for future reference and training.
• Keep abreast of new data engineering technologies and introduce innovative solutions.

Requirements

• BS or MS degree in Computer Science, Information Systems, Engineering, or equivalent professional experience.
• 5 years of Data Engineering experience with Amazon Web Services.
• 8+ years of experience in designing, implementing, troubleshooting, and monitoring complex production data pipelines or enterprise software solutions.
• 3 years of experience managing a team of software and/or data engineers.
• Expert programming skills in Python, CDK scripting, and SQL knowledge including stored procedures.
• Experience in designing and implementing complex ETL or ELT pipelines in relational and non-relational databases including Data Lakes.
• 3 years of experience developing and implementing Continuous Integration and Continuous Delivery (CI/CD) systems.
• 5+ years of experience developing and implementing test-driven architecture solutions.
• Experience supervising junior team members in project-based work or formally via managerial reporting.
• Expert in systems analysis, design, development, quality assurance, and integration methodologies.
• Excellent problem-solving, analytical, and troubleshooting skills.
• Excellent verbal and written communication skills.

Nice-to-haves

• 8 years of experience in Data Engineering or related field with at least 5 years of hands-on experience designing and building complex data pipelines using cloud-based infrastructure, preferably AWS Technologies.
• 5 years of experience or more with Agile software development methodologies and tracking software.
• 3+ years of experience managing teams.
• Experience with Data Lakes, Delta Lakes, Lakehouse, Spark, Hive, Apache Airflow, Testing Frameworks, Streaming Pipelines, Kafka.

Benefits

• Competitive health benefits (Employee premiums covered at 100%; family premiums at 50%)
• Vision, dental, life, and disability insurance options
• Paid vacation, sick leave, and holidays
• Teachers Retirement System of Texas (a defined benefit retirement plan)
• Additional voluntary retirement programs: tax sheltered annuity 403(b) and a deferred compensation program 457(b)
• Flexible spending account options for medical and childcare expenses
• Training and conference opportunities
• Tuition assistance
• Athletic ticket discounts
• Access to UT Austin's libraries and museums
• Free rides on all UT Shuttle and Capital Metro buses with staff ID card