Senior AI Product Manager, Evals & Annotations

Arize AI is the leading observability and evaluation platform for AI engineers building and deploying agents and applications. The company serves over 150 enterprise customers including Booking.com, Uber, and Siemens. Arize enables teams to monitor, troubleshoot, and optimize AI systems from prototype through production, bringing build, test, and run capabilities into a unified workspace.

As generative AI and autonomous agents reshape technology across industries, evaluation has become central to how developers measure and improve their systems. Arize is seeking a Senior Product Manager to own the Evals & Annotations product area end-to-end, driving strategy and execution for how teams define success, run evaluations, incorporate human feedback, and iterate on their AI applications.

What you'll do

• Own the complete evaluation lifecycle including eval creation and execution, human review, annotation workflows, and broader evaluation infrastructure.

• Set product strategy and roadmap for the Evals & Annotations area, balancing customer requirements, foundational platform needs, UX improvements, and emerging opportunities in autonomous and managed agent development.

• Independently develop product vision, build prototypes, and partner with Design and Engineering to shape intuitive, polished experiences that reflect your strong opinions about how evaluation workflows should function.

• Work closely with customers through multiple weekly conversations to understand their AI evaluation practices, identify underlying problems, and validate product direction.

• Collaborate with Engineering on technically complex challenges spanning evaluation execution, backend systems, orchestration, data infrastructure, and architectural tradeoffs.

• Drive products from problem definition through launch, adoption, and iteration while making clear prioritization decisions across competing demands.

• Partner directly with cofounders and cross-functional teams on product direction, design exploration, and prioritization in an environment where the founding team remains actively involved in product thinking.

What you'll bring

• Significant experience independently owning technically complex developer-facing products, with flexibility on tenure for candidates demonstrating exceptional technical depth and product judgment.

• Deep understanding of LLM evaluations and AI development workflows including automated evals, human feedback loops, experimentation, and how developers iterate on AI applications and agents.

• Strong technical background with developer tools, AI infrastructure, ML platforms, or adjacent technical products, plus solid understanding of backend systems and software architecture.

• Hands-on engineering ability including debugging independently, navigating code, building prototypes, submitting pull requests, and engaging in credible technical discussions with engineers about implementation approaches and tradeoffs.

• Excellent product and UX instincts with the ability to independently define and prototype core product experiences while partnering effectively with designers and design engineers.

• Strong ownership mentality, initiative, and bias toward action, comfortable navigating ambiguity and getting hands-on without waiting for fully defined direction.

• Collaborative approach to working with founders, engineers, and designers; you bring strong opinions, engage constructively, and thrive where responsibilities overlap.

• Experience working closely with technical customers and strong communication skills moving between strategy and execution.

Nice to have

• Direct experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration systems.

What they offer

• Remote position based in the United States.

• Full-time employment with a well-funded platform serving leading enterprises.