Senior Software Test Engineer
Explicitly requires vibe coding skills — the team uses AI coding tools (Claude Code, Cursor) as the primary development surface and relies on AI for evals and graders.
About the Role
Senior Software Test Engineer to build eval infrastructure and do hands-on exploratory testing for AI agents, enabling rapid, high-quality delivery by combining automation, production forensics, and close partnership with business stakeholders.
Job Description
Role
Curative is hiring a Senior Software Test Engineer to drive quality and measurement for AI agents and production services. This role combines building eval and observability infrastructure with hands-on exploratory testing, production forensics, and close collaboration with business teams.
Key Responsibilities
- Design, build, and maintain the eval platform: harnesses, datasets, graders, and CI integration to measure agent behavior (task success, tool-use correctness, drift, latency, cost).
- Perform hands-on exploratory testing across member, operations, and provider workflows, including unusual and edge paths.
- Validate that the delivered product matches stakeholder needs by working directly with business teams and extracting real workflows.
- Develop production signal, dashboards, and failure taxonomies; reproduce incidents, diagnose root causes, and fix or hand off precise remediation steps.
- Make pragmatic product and prioritization decisions under ambiguity; act as a de facto product/quality owner when needed.
Requirements
- 5+ years in software quality, testing, or engineering with experience in both automation and exploratory testing.
- Strong fluency with LLM evaluation: golden datasets, programmatic graders, LLM-as-judge approaches, and regression suites for prompts and agent loops.
- Hands-on experience with AI agents and common failure modes (silent drift, tool misuse, compounding errors).
- Deep debugging skills: logs, traces, queries, and production forensics, with ability to fix issues in Python or TypeScript.
- Strong stakeholder communication and the ability to extract requirements from non-technical partners.
- Comfort working without mature processes, making fast trade-offs, and reviewing AI-generated code diffs.
- Excellent written communication and pragmatic quality judgment.
Preferred
- Experience building eval or observability infrastructure from scratch.
- Familiarity with LLM observability tooling (LangSmith, Braintrust, Arize) or equivalent homegrown solutions.
- Experience defining agent guardrails and permissible unsupervised behaviors.
- Product experience (requirements, behavior definition, acting as de facto PM for a surface).
Why this role
You will be close to the business and own both the testing infrastructure and the product outcomes. The role offers wide responsibility across eval infrastructure, hands-on testing, production stakes, and direct collaboration with product and operations.
Benefits
- Curative Health Plan (100% employer-covered medical premiums for you and 50% coverage for dependents on the base plan)
- $0 copays and $0 deductibles (with completion of Baseline Visit)
- Preventive and primary care, mental health support, one-on-one care navigation, chronic condition programs
- Maternity and family planning support
- 24/7/365 Curative Telehealth
- Pharmacy, comprehensive dental and vision coverage
- Employer-provided life and disability coverage with supplemental options
- Flexible spending accounts
- Generous PTO policy and 11 paid company holidays
- 401(k) for full-time employees
- Up to 8–12 weeks paid parental leave (role eligibility-dependent)