Lead Engineer, AI

Circle is building an AI-powered platform that brings together creators, educators, and businesses with tools for community engagement, live events, courses, and payments all under one brand. The company operates as a fully remote organization with around 270 team members across 30+ countries, holding company offsites twice yearly in destinations around the world. For this role, Circle's R&D team is seeking a Lead Engineer to establish the foundations for measuring, diagnosing, and improving the quality of their AI-powered features in production environments.

The AI Quality engineering team at Circle focuses on infrastructure rather than model research or training. They need someone to drive evaluation frameworks, observability tooling, and diagnostic systems that determine whether AI agents perform reliably, identify improvement areas, and optimize for speed and cost efficiency.

What you'll do

• Design and maintain evaluation infrastructure including CI/CD pipelines, scorers, and datasets to assess whether Circle's AI agents (planners, tool-callers, and sub-agents) function correctly across single tool calls and multi-turn conversations.

• Pinpoint quality failures by tracing issues through the agent pipeline, including plan creation versus execution and tool selection across complex features like workflows and analytics, then prototype solutions or surface priorities for the core engineering team.

• Expand evaluation datasets through annotation workflows, AI-generated conversations, and simulations to cover more product surface area beyond manual labeling.

• Run structured experiments testing different prompts, models, and agent configurations, evaluate open-source and new models against production baselines, and optimize for cost and latency through model selection, caching, and complexity-based routing.

• Lead and develop the AI Quality engineering team, setting technical direction and managing priorities while remaining actively involved in coding.

• Collaborate with AI Core engineering to ensure product changes genuinely improve quality metrics.

What you'll bring

• Seven or more years shipping production software, preferably including LLM-powered agents that perform real actions, with experience on complex tool-using systems involving multiple tools, planning, orchestration, or sub-agents that you can discuss in detail.

• Proficiency in Ruby on Rails and Python, or the ability to learn them quickly; Rails powers Circle's production system and agent foundation, while Python skills are especially valuable for data and evaluation work.

• Background building evaluation or observability infrastructure for ML/AI systems, including experience with eval pipelines, scorers, dashboards, CI/CD for evaluations, and frameworks like Braintrust or LangSmith.

• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation, including knowledge of turning raw examples into trustworthy datasets through human annotation, AI-generated data, or simulation.

• Ability to learn quickly, experiment aggressively, and operate in dynamic environments where results inform decisions even when experiments fail.

• Comfort navigating fast-paced settings with ambiguity and picking up new technologies as projects demand.

• Professional-level English proficiency (CEFR Level C2 / ILR Level 5).

What they offer

• U.S.-benchmarked compensation with company equity including ongoing refresh grants.

• 35 days of paid time off annually.

• Remote-only work with company retreats twice yearly at international destinations, flexibility, autonomy, and a benefits package supporting health, wellbeing, and professional development.