Site Reliability Engineer

About This Role

This Site Reliability Engineer role focuses on campus reliability, involving the design of campus monitoring systems, technical command during cross-discipline SEVs, and building safeguards to mitigate future incidents. The position acts as a crucial link across compute, network, storage, power, and cooling systems within one of the world's largest AI training campuses. It requires calm incident leadership, sound judgment in fleet-scale observability, and the ability to drive reliability initiatives across both software and facility domains. The ideal candidate will have experience in power plants, nuclear power plants, data centers, or similar facilities with power, compute, cooling, and related plant systems.

Responsibilities

• Own monitoring architecture and signal quality across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling): defining what to alert on, suppress, and trust
• Consume NOC noise-disposition feedback to drive suppression and redesign, treating alert noise as a design failure
• Provide SEV command support: technical incident leadership, bridge coordination with the NOC, paging and escalation (e.g., Pager Duty), and timeline and severity hygiene
• Run blameless postmortems and drive corrective actions to closed status
• Gate high-risk change: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through a disciplined change management process
• Lead cross-functional reliability projects spanning compute, network (Infini Band/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing)
• Partner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs
• Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current
• Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects)
• Define error budgets and availability objectives at campus and service boundaries as adopted by the business
• Participate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus

Requirements

• Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field (or equivalent experience)
• 5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production
• Experience in power plants, nuclear power plants, data centers, or facilities like SpaceXSI — current or prior — spanning power, compute, cooling, or related plant systems
• Proven large-scale incident command experience and calm technical leadership on a bridge