Supercompute Operations Engineering

xAI is building advanced artificial intelligence systems to help humanity better understand the universe. The company operates a full stack of infrastructure and models, from large-scale supercomputers to applications serving millions of users. As a Supercompute Operations Engineering professional, you will join the Data Center team responsible for maintaining continuous GPU availability across xAI's distributed compute facilities.

xAI was founded to develop beneficial artificial intelligence aligned with human values. The company operates some of the world's largest GPU clusters, combining cutting-edge hardware with sophisticated software systems to train and deploy models at unprecedented scale. xAI's infrastructure supports both research and production workloads across multiple geographic locations.

What you'll do

• Design and maintain monitoring systems for GPU and host telemetry using tools like DCGM and Prometheus/Grafana, as well as facility monitoring through BMS, EPMS, DCIM, and SCADA platforms, ensuring comprehensive visibility across all systems.

• Manage the incorporation of new compute hardware including physical, firmware, and software updates across the fleet.

• Participate in 24/7 incident response and serve as control authority from customer impact through facility operations, with responsibility for mitigation and resolution across all layers.

• Execute day-to-day site sustainment operations, triaging and resolving issues affecting rack-level compute and facility infrastructure.

• Lead acceptance testing and deployment campaigns to bring new data halls into production.

• Build production automation for new data hall deployment, including automated alerting and incident response systems.

• Develop and maintain standard operating procedures and technical documentation for site and facility technicians, incorporating equipment changes and upgrades.

• Lead cross-functional reliability initiatives spanning compute, network infrastructure, storage, and facility systems, particularly addressing failures that cross traditional system boundaries.

• Analyze fleet-wide data including mean time between failures and failure rates to identify systemic issues and drive proactive alerting, vendor responses, and supplier improvements.

What you'll bring

• Bachelor's degree in Mechanical, Electrical, Computer Engineering or related scientific field, plus 2+ years of hands-on data center, critical facilities, or large-scale compute operations experience; or 6+ years of equivalent experience.

• Hands-on expertise with Prometheus/Grafana, NVIDIA DCGM, or industrial control systems such as SCADA, BMS, EPMS, or DCIM.

• Experience with site reliability, systems engineering, plant operations, or large-scale production environments.

• Proficiency in scripting languages like Python or Bash, plus experience in at least one systems language such as C, C++, Java, Go, or Rust.

• Demonstrated ability to lead incident response with calm technical judgment and coordinate across distributed, multi-disciplinary teams.

• Ability to write, maintain, and operationalize playbooks, runbooks, and standard operating procedures at scale.

Nice to have

• Experience with AI/ML infrastructure or supercomputing environments, including large-scale GPU fleets and tools like NCCL, InfiniBand, RoCE, Slurm, or Kubernetes.

• Fleet firmware and BMC management using Redfish, IPMI, and staged firmware rollouts.

• Background running acceptance test campaigns for integrated systems.

• Familiarity with data center hardware signals including servers, GPUs, networking, power, and cooling systems.

What they offer

• Full-time position based in Southaven, Mississippi or Memphis, Tennessee with relocation support available.

• Opportunity to work on infrastructure powering next-generation artificial intelligence at global scale.