Heavy use of LLMs and AI coding tools to build agentic systems; role centers on building and evaluating LLM-based agents and rapid prototyping with AI.
About the Role
Lead the design and implementation of an Agentic SOC that uses LLM-based agents to investigate security signals end-to-end, augmenting SOC analysts by automating high-volume, repetitive work while preserving security, auditability, and human handoffs. Own system architecture, reliability, evaluation, and technical leadership for production-grade agent behavior and decision-making.
Job Description
Role
Huntress is hiring a Principal Software Engineer to lead the design and delivery of an Agentic Security Operations Center (SOC). The role owns the architecture and technical direction of an LLM-based agent system that investigates signals end-to-end, assembles evidence, and decides when to act autonomously versus hand off to a human analyst. You will work closely with SOC analysts, detection engineers, product, and other engineering teams and drive production-grade reliability, auditability, and tenant isolation.
Key Responsibilities
- Own the architecture of agentic investigations: signal ingestion, investigation state management, tool surfaces for agents, and final investigation artifacts (verdict, timeline, scope, evidence).
- Design and build LLM-powered agents that replicate strong analyst behavior and make appropriate decisions about escalation and handoff.
- Build LLM-based pre-processing to summarize, correlate, and surface what matters from analyst tools and telemetry.
- Embed a security mindset in the agent design: trust boundaries, tenant isolation, attack resistance, and full audit trails.
- Define and own evaluation: labeled datasets, offline evaluation, regression gates, and metrics to measure non-deterministic system quality over time.
- Decide, based on data, what the system can act on autonomously and what requires human review; evolve that decisioning responsibly.
- Engineer for production: reliability, latency, cost at scale, and robustness to model failures or changes.
- Prototype and derisk uncertain approaches, then hand off production-ready designs to the team.
- Represent the team technically across SOC leadership, product, other engineering teams, and executives; drive alignment and protect team scope.
- Mentor engineers, perform thoughtful code review, and raise both the team’s baseline and ceiling through patterns and tooling.
Requirements
- 15+ years building complex software products with significant time as a technical lead on production systems.
- Hands-on experience designing, shipping, and operating LLM-based agents in production (tool use, context management, structured output, guardrails, evaluation) and an understanding of failure modes.
- Experience building evaluation frameworks for non-deterministic systems (labeled datasets, offline evals, regression gates).
- Security mindset with experience handling untrusted input, multi-tenant isolation, and auditable trails.
- Experience building systems that make automated decisions with real consequences and sound judgment about what to automate.
- Deep backend expertise in distributed systems, queues, durable workflows, and concurrency; designing for throughput and correctness under load.
- Strong skills in one or more backend languages and ability to learn new ones quickly; primary stack is Ruby on Rails and fluency is expected.
- Experience with public cloud environments (AWS, Azure, or similar) and data stores such as Postgres and Redis.
- Familiarity with AI coding tools (example: Claude Code).
- BS or MS in Computer Science/Engineering or equivalent experience.
Bonus Qualifications
- Experience in SOCs, incident response, threat hunting, or detection engineering.
- Familiarity with endpoint and identity telemetry and attacker tradecraft.
- Experience adversarially testing ML/LLM systems.
- Additional experience with Ruby on Rails.
Compensation & Location
- Base: $215,000 to $240,000 (plus bonus and equity).
- Location: Remote (U.S. only).
Benefits
Huntress provides 100% remote work, generous PTO (vacation, sick, holidays), 12 weeks paid parental leave, comprehensive medical/dental/vision plans, 401(k) with 5% contribution regardless of employee contribution, life and disability insurance, stock options for full-time employees, $500 one-time home office reimbursement, annual education/professional development allowance, $75/month digital reimbursement, and access to coaching (BetterUp).