Senior Production Engineer

• At Crusoe, our Production Engineering team plays a pivotal role in ensuring the reliability and performance of our infrastructure. Production Engineering at Crusoe is dedicated to detecting, analyzing, and preventing issues to maintain high Service Level Agreement through Service Level Indicators (SLIs) and Service Level Objectives (SLOs)
• Through automation and proactive remediation, our Production Engineers not only resolve common errors automatically but also advise various engineering teams in building resilient code
• We prioritize anticipating and resolving issues before they impact our customers, conducting thorough post-mortems, and driving continuous improvement
• Our customer-centric approach ensures that clients always have access to the virtual machines they depend on
• Join us to help build and maintain the robust systems that power Crusoe’s innovative solutions
• As a Production Engineer at Crusoe, your day begins with a review of overnight alerts and system performance metrics to ensure everything is running smoothly
• We operate close to the hardware, so many of these will be novel hardware failures on cutting edge equipment that are challenging to diagnose and automate
• You will collaborate with your team in a morning stand-up meeting to discuss ongoing projects, recent incidents, and priorities for the day
• Your tasks might include automating routine processes, analyzing system logs, and developing tools to enhance our monitoring capabilities
• You’ll spend part of your day working closely with software engineers, advising on best practices for resilient code and reviewing changes before deployment
• Regularly, you will engage in incident response drills, post-mortems, and root cause analysis sessions to learn from past issues and prevent future ones
• Throughout the day, you will stay focused on maintaining high SLIs and SLOs, ensuring that our infrastructure remains robust and reliable for our customers
• By day’s end, you will document your work, share insights with your team, and plan for the next day’s challenges, always with a customer-centric mindset

Benefits

• Health & wellbeing: Comprehensive health benefits designed to support your overall wellness
• Time away: Paid time off for vacations, family bonding, and unexpected needs
• 401(k) match: Build your financial future with our 401(k) matching program
• Mental wellness: Resources and support for your emotional wellbeing and navigating life’s challenges- Excellent communication skills
• 5+ years of professional production engineering experience
• Solid understanding of infrastructure design, including the operational trade-offs of various designs
• 3+ years of experience contributing to architecture and design (architecture, design patterns, reliability and scaling) of new and current systems
• Experience with Unix/Linux environments
• Experience writing high quality code with at least one programming language (Python, Go, or similar)
• Bachelor’s Degree in Computer Science or related field, or 8+ years relevant work experience
• Experience with information security best practices
• Experience with TCP/IP and network programming
• Experience building with modern infrastructure tools such as Docker, Kubernetes, Ansible, Cloud Formation, Terraform
• Experience building with modern CI/CD practices and build systems, such as GitLab CI/CD, CircleCI, GitHub Actions
• Experience with logging, monitoring and alerting systems and tools
• Excited to work with cutting edge AI training & inference hardware and networks
• Experience running large, mission-critical storage systems / NVMe over Fabric