Senior Software Engineer II Applied AI and Evaluations

About the Role

Smartsheet is building the next generation of AI-powered work management through SmartAssist, an intelligent agent platform. As the platform scales to production-grade agents, the team requires an expert to own the technical quality frontier.

This is a high-autonomy position at the intersection of LLM evaluation, prompt and context engineering, and retrieval-augmented generation (RAG). You will diagnose agent failures, design systems to catch regressions, and drive measurable improvements across the orchestrator and subagent fleet.

• Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagents.
• Identify failure modes across quality dimensions including factual accuracy, completeness, tone, actionability, and latency.
• Drive quality improvements through advanced prompt engineering.
• Work closely with Agent Engineering and AI Platform teams within a mature Agent Development Lifecycle (ADLC).
• Utilize evaluation infrastructure built on Databricks and MLflow.

Required Skills