Staff Machine Learning Engineer, LLM Inference Optimization
Nebius
About the position
Join a leading tech startup focused on building a full-stack AI cloud platform, optimizing model inference and deployment.
Tech stack
- python
- pytorch
- tensorrt-llm
- triton
- cuda
Requirements
Required:
- Strong Python and PyTorch engineering skills
- Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference systems
- Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, or equivalent internal systems
- Strong understanding of transformer inference bottlenecks, including KV cache, attention, memory bandwidth, batching, parallelism, and long-context serving
Nice to have:
- Experience with quantization-aware training, post-training quantization, FP8, INT8, INT4, NVFP4, MXFP4, AWQ, GPTQ, SmoothQuant, or related techniques
- Experience with distillation, speculative decoding, EAGLE, Medusa, multi-token prediction, or other inference acceleration methods
- Experience with agentic workloads, including tool calling, structured outputs, streaming APIs, high concurrency, and multi-step orchestration
Responsibilities
- Own a platform-level optimization pillar such as quantization, speculative decoding, KV-cache optimization, engine selection, or long-context serving
- Create reusable evaluation, benchmarking, and rollout frameworks across multiple model families
- Provide technical direction to MLEs, Applied Scientists, GPU kernel engineers, and infrastructure engineers on shared performance goals
- Optimize LLM and VLM endpoints for latency, throughput, memory efficiency, GPU utilization, quality, and cost per token
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture