Program Lead: Product Operations - AI Observability

Program Lead: Product Operations - AI Observability Department: Community Operations Team: Strategy & Program Operations Location: Sunnyvale, United States Type: Full-Time **About the Role** The **AI Observability Program Leader** will own the end-to-end strategy, design, and implementation of the frameworks used to monitor, understand, and improve Uber’s GenAI-powered agentic systems. This role sits within the Global Digital Experience team, the operational arm of Uber’s customer support tech organization, and is a critical driver of accuracy, safety, and reliability across Uber’s next-generation AI solutions. This leader will bridge the gap between raw AI logs and actionable product insights. You will define the methodologies for **agentic reasoning observability**, develop **automated evaluation (autoeval) systems**, and design **simulators** to stress-test AI performance before it reaches the customer. You will partner closely with Product, Engineering and Data Science to translate complex agent behaviors into **micrometrics**—the granular signals that help us pinpoint exactly where a reasoning chain succeeded or failed. The ideal candidate brings a systems thinking mindset, technical literacy in LLM orchestration, and the ability to influence technical roadmaps through rigorous data and observability frameworks. **What You'll Do** - **Architect Observability Frameworks:** Own the strategy for understanding AI agentic reasoning, enabling deep analysis of step-by-step agent decision-making. - **Drive Autoeval Strategy:** Design and roll out automated evaluation systems (LLM-as-a-judge) to provide a scalable, high-confidence "pulse" on AI performance across conversational and voice interfaces. - **Define Micrometrics:** Develop granular signals within agentic activity—identifying latent failures, reasoning loops, or tool-calling inefficiencies—to drive product improvements - **Lead Pre-Launch Simulation:** Partner with Product & Engineering to build and maintain simulation environments that test AI agents against edge cases before deployment, and democratise these tools with Operations teams - **Cross-Functional Technical Partn