Principal Engineer - Observability Solutions

About the position

At T-Mobile, we invest in YOU! Our Total Rewards Package ensures that employees get the same big love we give our customers. All team members receive a competitive base salary and compensation package - this is Total Rewards. Employees enjoy multiple wealth-building opportunities through our annual stock grant, employee stock purchase plan, 401(k), and access to free, year-round money coaches. That's how we're UNSTOPPABLE for our employees! We are seeking a Principal Engineer - Observability Solutions to drive the design, development, and delivery of end-to-end observability capabilities across T-Mobile's technology ecosystem. This hands-on technical leadership role will focus on creating scalable, resilient, and intelligent observability solutions that empower teams to gain real-time visibility into application and infrastructure health, performance, and user experience. The Principal Engineer - Observability Solutions will collaborate closely with software engineers, site reliability engineers, infrastructure teams, and business stakeholders to architect systems that capture, process, and analyze telemetry data (metrics, logs, traces, events). The ideal candidate is passionate about driving operational excellence, experienced in modern observability stacks, and comfortable influencing at both technical and strategic levels. The Principal Engineer - Observability Solutions works on challenging technical assignments that require sound judgment in issue resolution and decision-making. They determine methods and procedures for new assignments and handle tasks that demand significant initiative and strategic thinking. They understand the broader implications of their work and provide recommendations for solutions. Typically, they receive minimal instruction on daily tasks and only general guidance for newly introduced assignments. In some cases, they may serve as an informal team leader. We pride ourselves on encouraging a culture of innovation, advocating for agile methodologies, and promoting transparency in all that we do. Join us in embodying the spirit of the 'Un-carrier' and make a tangible impact!

Responsibilities

• Architect and implement observability platforms and tools that support proactive detection, root cause analysis, and continuous improvement across distributed systems.
• Lead the design and development of core observability services, including metrics pipelines, log aggregation, tracing infrastructure, alerting systems, and data visualization.
• Leverage modern technologies such as Open Telemetry, cloud-native instrumentation, streaming data platforms, and AI/ML for anomaly detection and automated insights.
• Collaborate with engineering and SRE teams to define service-level objectives (SLOs), error budgets, and golden signals.
• Provide technical leadership and mentorship to engineering teams, ensuring best practices in system design, telemetry standards, and observability instrumentation.
• Champion observability as a first-class concern in the software development lifecycle.
• Influence platform strategy and roadmap through deep technical insight and alignment with business priorities.
• Write advanced documentation, architecture, capabilities, limitations, and advantages for technologies.
• Monitor and influence relevant industry technologies and standards.
• Evaluate new and existing technology options that support business goals.

Requirements

• Bachelor's Degree in Engineering or Computer Science (Required)
• 7-10 years of technical engineering experience (Required)
• Self-motivated and able to work well under pressure with minimal supervision.
• Ability to work cooperatively with others on a team.
• Clear business and technical writing skills.
• Ability to demonstrate concern and meet external and/or internal customers' needs.
• Ability to tackle a problem using a logical, systematic, sequential approach.
• Depth of knowledge and skill in a technical area.
• Ability to anticipate implications and consequences of situations.
• Ability to gain others' support for ideas, proposals, projects, and solutions.
• Ability to enhance others' commitment to their work.
• Ability to convey confidence in employees' ability to be successful.
• Ability to find effective solutions by taking a holistic perspective.
• Ability to analyze the organization's competitive position.

Benefits

• Medical, dental and vision insurance
• Flexible spending account
• 401(k)
• Employee stock grants
• Employee stock purchase plan
• Paid time off and up to 12 paid holidays
• Paid parental and family leave
• Family building benefits
• Back-up care
• Enhanced family support
• Childcare subsidy
• Tuition assistance
• College coaching
• Short- and long-term disability
• Voluntary AD&D coverage
• Voluntary accident coverage
• Voluntary life insurance
• Voluntary disability insurance
• Voluntary long-term care insurance
• Mobile service & home internet discounts
• Pet insurance
• Access to commuter and transit programs