Lead Data Engineer

$130,000 - $143,712/Yr

The University of Texas System - Austin, TX

Full-time - Mid Level

Remote - Austin, TX

Educational Services

About the position

The Lead Data Engineer for the UT Data Hub plays a crucial role in enhancing university outcomes by improving the usability and value of institutional data. This position involves leading a team of data engineers to create complex data pipelines within the university's cloud data ecosystem, supporting both academic and administrative needs. The role emphasizes collaboration, creativity, and effective communication to solve technical challenges and drive data-informed decision-making across the university.

Responsibilities

• Design, plan, develop and deploy production quality enterprise level data pipelines using Cloud technologies.
• Conduct and participate in code reviews to ensure high-quality code.
• Implement industry best practices for data engineering and process improvements.
• Identify and address bottlenecks in data pipelines, focusing on optimizations and scalability.
• Develop systems to ensure data quality, consistency, and security compliance.
• Contribute to coding, debugging, and solving complex engineering challenges.
• Lead technical knowledge share sessions discussing current industry trends in Data Engineering.
• Define project milestones, timelines, and deliverables for on-time delivery.
• Collaborate with internal and external stakeholders, including Data or System Architects and business users.
• Manage, prioritize, and track team and project work while balancing urgent requests.
• Communicate and manage risks, roadblocks, and resource allocations for project delivery.
• Supervise a team of data engineers, providing guidance and mentorship.
• Participate in hiring and recruitment activities for new team members.
• Conduct regular performance reviews and set development goals for team members.
• Promote collaboration within the team and across departments for project execution.
• Provide regular updates on project status and technical challenges to various audiences.
• Document all project deliverables for future reference and training.
• Stay updated with new data engineering technologies and introduce innovative solutions.

Requirements

• BS or MS degree in Computer Science, Information Systems, Engineering, or equivalent professional experience.
• 5 years of Data Engineering experience with Amazon Web Services.
• 8+ years of experience in designing, implementing, troubleshooting, and monitoring complex production data pipelines or enterprise software solutions.
• 3 years of experience managing a team of software and/or data engineers.
• Expert programming skills in Python, CDK scripting, and SQL including stored procedures.
• Experience in designing and implementing complex ETL or ELT pipelines in relational and non-relational databases including Data Lakes.
• 3 years of experience developing and implementing Continuous Integration and Continuous Delivery (CI/CD) systems.
• 5+ years of experience developing and implementing test-driven architecture solutions.
• Experience supervising junior team members in project-based work or formally via managerial reporting.
• Expert in systems analysis, design, development, quality assurance, and integration methodologies.
• Excellent problem-solving, analytical, and troubleshooting skills.
• Excellent verbal and written communication skills.

Nice-to-haves

• 8 years of experience in Data Engineering or related field with at least 5 years of hands-on experience designing and building complex data pipelines using cloud-based infrastructure, preferably AWS Technologies.
• 5 years of experience or more with Agile software development methodologies.
• 3+ years of experience managing teams and implementing software development processes.
• Experience with Data Lakes, Delta Lakes, Lakehouse, Spark, Hive, Apache Airflow, Testing Frameworks, Streaming Pipelines, Kafka.

Benefits

• Competitive health benefits (Employee premiums covered at 100%; family premiums at 50%)
• Vision, dental, life, and disability insurance options
• Paid vacation, sick leave, and holidays
• Teachers Retirement System of Texas (a defined benefit retirement plan)
• Additional voluntary retirement programs: tax sheltered annuity 403(b) and a deferred compensation program 457(b)
• Flexible spending account options for medical and childcare expenses
• Training and conference opportunities
• Tuition assistance
• Athletic ticket discounts
• Access to UT Austin's libraries and museums
• Free rides on all UT Shuttle and Capital Metro buses with staff ID card