Own the inference data plane for custom ML accelerators at Amazon Annapurna Labs

Build and optimize the software that runs large language models on Amazon's custom ML accelerator hardware, from compute kernels through serving-framework integration. This role sits at the intersection of ML systems and low-level performance engineering, working end-to-end from PyTorch model definitions down to distributed execution on custom silicon.

Practice this interview

Free · a live voice mock calibrated to this exact role

Start the mock interview

What this interview tests

Compute kernel development for custom ML accelerators
LLM architecture validation (decoder-only, MoE) end-to-end
Integrating custom hardware backends into vLLM/PyTorch serving frameworks
Inference profiling and performance optimization
Test/CI infrastructure for model correctness across hardware targets
C/C++ and Linux systems fundamentals

Common question themes

Design a compute kernel for a custom accelerator and how you'd validate it end-to-end

Debugging a throughput or latency regression from simulation to hardware

Integrating a new hardware backend into vLLM: scheduler, memory management, model parallelism

KV cache optimization or speculative decoding tradeoffs

Building CI/CD gates for correctness and performance regressions

Mentoring and driving design reviews on a fast-moving hardware/software team

How candidates describe it

Real Senior ML Software Engineer interview stories — retold from candidates' public write-ups, with sources.

Amazon · Applied Scientist (L4)Offer

Amazon Applied Scientist Interview Experience: Alexa Speech Team, 2021

A redirected recruiter call turned into an Amazon Applied Scientist loop with the Alexa Speech team: a phone screen, a split five-round virtual onsite across two teams, a bar raiser, and an added ML-breadth round, ending in an offer with a downlevel from L5 to L4.

Interviewed July 2021 · Boston, MA (Remote)

Google · L5 Machine Learning EngineerOffer

Google L5 machine learning engineer interview: phone screen skipped, four technical rounds, and an offer

A candidate applying for an L5 machine learning engineer role at Google had the phone screen skipped due to a referral and prior tenure at the company, then went through four technical and design rounds plus a behavioral round before receiving an L5 offer. The loop was one leg of a broader search that produced offers from several companies in the same cycle.

Interviewed April 2022 · Remote

View the original posting

All Amazon interviews