Senior Site Reliability Engineer - Observability and Telemetry Platform

$148,000 - $276,000/Yr

Nvidia - Santa Clara, CA

Full-time - Mid Level

Santa Clara, CA

Computer and Electronic Product Manufacturing

About the position

Site Reliability Engineering (SRE) at NVIDIA is an engineering discipline focused on designing, building, and maintaining large-scale production systems with high efficiency and availability. This role combines software and systems engineering practices to ensure that our internal and external GPU cloud services operate with maximum reliability and uptime. SRE at NVIDIA is not just a technical role; it embodies a mindset and a set of engineering approaches aimed at optimizing production systems. The discipline requires a deep understanding of various systems, networking, coding, database management, capacity management, continuous delivery, and deployment, as well as familiarity with open-source cloud technologies like Kubernetes and OpenStack. The SRE team is responsible for the overall health of our services, ensuring that they run smoothly and efficiently. This involves careful planning and preparation to enable developers to make changes to existing systems while monitoring capacity, latency, and performance. Our software development efforts are heavily focused on automation, performance tuning, and enhancing the efficiency of production systems. SREs play a crucial role in minimizing manual work and proactively identifying potential outages, which is essential for maintaining product quality and ensuring a dynamic work environment. At NVIDIA, we foster a culture of diversity, intellectual curiosity, and problem-solving. We believe that bringing together individuals with varied backgrounds and experiences leads to innovative solutions. Our team encourages collaboration, big thinking, and risk-taking in a blame-free environment. We prioritize self-direction in meaningful projects while providing the necessary support and mentorship for personal and professional growth.

Responsibilities

• Design, implement and support operational and reliability aspects of large scale Observability & Telemetry collection platform with a focus on performance at scale, real-time monitoring, logging and alerting.
• Engage in and improve the whole lifecycle of services—from inception and design through deployment, operation and refinement.
• Support services before they go live through activities such as system design consulting, developing software tools, platforms and frameworks, capacity management and launch reviews.
• Maintain services once they are live by measuring and monitoring availability, latency and overall system health.
• Scale systems sustainably through mechanisms like automation, and evolve systems by pushing for changes that improve reliability and velocity.
• Practice sustainable incident response and blameless postmortems.
• Be part of an on-call rotation to support production systems.

Requirements

• BS degree in Computer Science or a related technical field involving coding (e.g., physics or mathematics), or equivalent experience.
• 5+ years of experience with Infrastructure automation, distributed systems design, experience with design, develop tools for running large scale private or public cloud system in Production.
• 5+ years experience delivering foundational infrastructure and observability platforms.
• Experience in one or more of the following: Python, Go, Perl or Ruby.
• In-depth knowledge on Linux, Networking and Containers.

Nice-to-haves

• Interest in crafting, analyzing and fixing large-scale distributed systems.
• Systematic problem-solving approach, coupled with strong communication skills and a sense of ownership and drive.
• Ability to debug and optimize code and automate routine tasks.
• Experience in using or running large private and public cloud systems based on Kubernetes, OpenStack and Docker.
• Experience running Grafana, OpenTelemetry, Prometheus, and similar observability focused tools.

Benefits

• Equity and benefits eligibility based on location and experience.