What it takes to run agentic workflows in production

An agent that works in a notebook is a prototype. An agent that sends a patient a text message at 9:02 a.m. on a Tuesday is a production system — and it needs to be treated like one.

Start with the action, not the model

The most useful question isn't "which model?" but "what is this system allowed to do?" Enumerate the actions, give each one a typed contract, and make the model choose among them.

import { z } from "zod";
 
export const tools = {
  draftReply: {
    description: "Draft a reply to a patient message for staff review.",
    input: z.object({
      messageId: z.string(),
      intent: z.enum(["book", "reschedule", "question", "opt-out"]),
    }),
    // Drafts are never sent automatically.
    requiresApproval: true,
  },
} as const;

Keep a human on the irreversible steps

Anything that can't be undone — sending, charging, deleting — goes through an approval step. It's slower, and it's the reason people trust the system enough to use it.

Make every run inspectable

Log the inputs, the tool calls and their results for every run, keyed by a single trace id. When something goes wrong (it will), you want to replay the exact run rather than guess.

Evaluate like you mean it

A small, hand-labelled set of real cases beats a large synthetic one. Run it on every prompt or model change, and track the regressions, not just the average.