Amazon
Senior
Own the inference data plane for custom ML accelerators at Amazon Annapurna Labs
Build and optimize the software that runs large language models on Amazon's custom ML accelerator hardware, from compute kernels through serving-framework integration. This role sits at the intersection of ML systems and low-level performance engineering, working end-to-end from PyTorch model definitions down to distributed execution on custom silicon.
Step into this interview
Free · a live voice mock calibrated to this exact role
What this interview tests
- Compute kernel development for custom ML accelerators
- LLM architecture validation (decoder-only, MoE) end-to-end
- Integrating custom hardware backends into vLLM/PyTorch serving frameworks
- Inference profiling and performance optimization
- Test/CI infrastructure for model correctness across hardware targets
- C/C++ and Linux systems fundamentals
Common question themes
Design a compute kernel for a custom accelerator and how you'd validate it end-to-end
Debugging a throughput or latency regression from simulation to hardware
Integrating a new hardware backend into vLLM: scheduler, memory management, model parallelism
KV cache optimization or speculative decoding tradeoffs
Building CI/CD gates for correctness and performance regressions
Mentoring and driving design reviews on a fast-moving hardware/software team
Related interviews
Amazon
Mid
Software Development Engineer II, Amazon Smart Vehicles
Amazon
Mid
Software Development Engineer, Ads AI Core Infra
Amazon
Mid
Software Engineer II, Leo Regulus
Cohere
Staff
Software Engineer, GPU Infrastructure (HPC)
Mid
Software Engineer
Mid