Cohere
Mid
Evaluate and debug AI coding-agent outputs to sharpen Cohere's models
Cohere is hiring a part-time independent contractor (16+ hrs/week, Canada-based, 12-month contract) to review and debug code, navigate repository architecture, and analyze coding-agent trajectories for model training and evaluation. The role wants 3-5 years of software engineering experience with proficiency across Python plus Java, JavaScript, Go, and SQL, prior API design/build/deploy experience, and ideally hands-on exposure to code agents like Claude Code, Cursor, Codex, or OpenCode — this is evaluation/annotation work, not product engineering.
Step into this interview
Free · a live voice mock calibrated to this exact role
Likely format
Initial resume/writing-sample screening, then a virtual annotation test (coding take-home + writing sample), then a video screen with the Operations team, then offer.
What this interview tests
- Multi-language code review (Python, Java, JavaScript, Go, SQL)
- Debugging and evaluating AI-generated code / agent trajectories
- API design critique
- Repository architecture navigation
- Precision and rigor in labeling/evaluation judgments
Common question themes
Review this code snippet or diff and identify what's wrong with it
Walk through evaluating whether a coding agent's trajectory actually solved the task correctly
Critique an API design — what's missing or wrong given the stated requirements
Describe a time you caught a subtle bug that looked correct on the surface
How do you navigate an unfamiliar codebase quickly to assess whether a change is sound
What's your experience using coding agents like Claude Code, Cursor, or Codex, and where have you seen them fail
Related interviews
Cohere
Mid
Forward Deployed Engineer, Agentic Platform
Cohere
Mid
Member of Technical Staff, Applied ML
Cohere
Senior
Data Engineer, Data Foundations
Airbnb
Mid
Software Engineer, Quality Platform
Lyft
Senior
Senior AI Software Engineer, Risk - Insurance Claims Management
Coinbase
Mid