About the role
Job Description Staff Software Engineer - AI Inference
Location: New York, NY (On-site, Remote)
About Syllo Syllo is a unified litigation platform enabling lawyers to use AI throughout the litigation lifecycle. We serve enterprise customers including major law firms and corporations.
About The Role We are building our own inference stack for open-source language models. You will define architecture, evaluate technologies, and build production systems for model serving, optimization, and monitoring.
Responsibilities
- Lead design and development of production inference platform
- Define technical roadmap for inference infrastructure and model serving
- Build scalable, cost-effective systems for LLM serving
- Evaluate and integrate inference technologies and runtimes
- Optimize latency, throughput, GPU utilization, memory, cost
- Develop deployment, routing, autoscaling, observability systems
- Partner with ML engineers to productionize models
- Establish benchmarking methodologies
- Make architectural decisions (build vs. leverage OSS/commercial)
- Mentor engineers and establish AI infrastructure best practices
Qualifications
- Significant experience with production AI inference systems
- Leadership in production LLM serving infrastructure
- Deep experience with inference runtimes (vLLM, SGLang, TensorRT-LLM, Triton, TGI, Dynamo, etc.)
- Strong background in distributed systems, backend infrastructure, or high-performance platform engineering
- Experience optimizing GPU inference performance (latency, throughput, batching, memory)
- Production GPU infrastructure experience
- Proficiency in Python and systems programming (Go, Rust, C++)
- Proven ability to lead technical architecture for complex initiatives
- Excellent communication and cross-team influence skills
Compensation Salary Range: $190,000 - $230,000 USD (plus health insurance and equity)
Remote: United States
Millions of jobs, with real people getting hired every day
Questions, answered
Click "Apply with JobAssist" – we tailor your resume and application to this role and submit it for your approval.
Yes. This role at ChatGPT Jobs was screened before publishing – we confirmed the employer before listing it.
The employer didn't disclose a salary range for this listing. JobAssist shows pay whenever it's available.
This position can be done from anywhere, with no in-office requirement.
Yes – every application is tailored from your profile and this job's requirements, and you can review and edit before it's sent.