Directly involved in evaluating and improving AI coding tools (Cursor, Claude Code, Copilot) and building AI dev tooling; hands-on with LLM prompting and model-inference tradeoffs.
About the Role
Join Tetriz as a DS/ML Engineer to build evaluation, routing, scoring, and MLOps systems that measure and improve how AI coding tools are used; work end-to-end across models, data pipelines, and product code to ship reliable, production-quality ML features.
Job Description
Role
Tetriz is hiring a DS/ML Engineer to build and maintain the evaluation and production infrastructure that underpins AI-native developer tools. The role centers on treating ML systems as a craft while contributing across product, backend, frontend, and data pipeline work to deliver reliable, low-latency features.
Key Responsibilities
- Design and implement evaluation systems for AI features: failure taxonomies, LLM-as-judge rubrics, golden datasets, and calibration against human judgment.
- Develop model routing strategies and run experiments to balance cost, quality, and latency; monitor regressions and economics of inference.
- Turn noisy real-world signals into trustworthy scores with statistical rigor; build scoring, calibration, and signal-quality systems.
- Contribute to MLOps and production lifecycle: feature pipelines, model versioning, rollout strategies, monitoring for drift, and ensuring reliable low-latency serving.
- Collaborate with engineering and product teams; ship backend or frontend code when required and help untangle data pipelines.
Requirements
Must have
- 1.5–2 years of hands-on experience in Data Science, Machine Learning, Software Engineering, or a related role.
- Experience building and shipping DS/ML systems end-to-end (professional work, projects, research, or OSS), not limited to notebooks.
- Comfort with Python and working knowledge of SQL.
- Basic grounding in applied statistics and the ability to reason about metrics and their pitfalls.
- Product-oriented “builder” instinct and willingness to work across layers (modeling, backend, frontend).
- Some exposure to LLMs (prompting, using APIs, or experimenting with model behavior).
Nice to have
- Experience with evaluation or observability tooling for LLM features.
- Experience with information retrieval, entity-matching, or record-linkage.
- Interest in developer-productivity, code analytics, or developer-experience data.
Notes
- The team emphasizes practical, production-focused ML engineering over pure research. The role expects curiosity across product and engineering domains and a willingness to do varied work to ship product-quality systems.