Human-in-the-Loop Engineering
A 2-hour hands-on workshop where an AI coding agent builds a webhook receiver from a flawed PRD, and the room decides every moment that matters.
This was not a talk to sit and watch. For two hours in Sofia, Stefan Angelov and I sat with a room full of engineers while Claude Code built a real, production-shaped service from a real, deliberately flawed PRD — and we stopped the agent at every point where a human, not the model, owned the decision. The question we kept returning to: the agent is genuinely capable, but capable of what, decided by whom?
We’d already written at length about why full automation is the wrong goal for human-in-the-loop systems — this workshop was that argument made concrete, with a keyboard in front of the room instead of a slide.
The scenario
Maria K. is the Payments PM at a fictional company. Her PRD asks engineering to fix a broken webhook integration with a payment provider called PayHub. The brief has every flaw a real PRD has: silent assumptions, an internal contradiction, deferred decisions, and a three-week timeline nobody had pressure-tested. It names concrete pain — support tickets from customers who paid but show as unpaid, a finance reconciliation discrepancy, a burst of roughly 5,000 events in five minutes whenever PayHub has an outage of its own — and three open questions it explicitly declines to answer: how long to retain raw payloads, whether failures need on-call paging or a daily digest, and whether out-of-order delivery matters.
We handed that PRD straight to Claude Code. Nothing about the PRD’s flaws was staged — they’re the ordinary condition of a real brief, and the exercise was to watch an agent meet them honestly rather than paper over them.
Architecture, decided before a line of code
One decision was pre-made and handed to the room as a constraint, not a question: the webhook receiver splits into a hot path, a warm path, and a cold path. The hot ingress path is pure Java 21 with no framework, built for a sub-five-second acknowledgement and sustained throughput. Processing, retries, and persistence sit on a warm path running Spring Boot 3.x, where developer velocity matters more than raw latency. Admin, reconciliation, and dead-letter review sit on a cold path, same reasoning again.
Left alone, Claude Code would default to Spring Boot everywhere — it’s the Java default, and defaults are exactly what an agent reaches for under velocity pressure. The first thing we wrote into the project’s CLAUDE.md forbids that default on the hot path. It’s the workshop in miniature: the human’s domain knowledge overriding the agent’s pattern-match, written down before the agent ever got a chance to guess.
The cast: eight agents, each refusing to decide alone
Rather than one general-purpose agent, the repo ships eight role-shaped subagents — product-owner, project-manager, solution-architect, backend-developer, devops-engineer, qa-lead, qa-engineer, and code-reviewer. A general-purpose agent will silently make decisions that should never be silent: which data store, what SLA, whether a feature is actually tested. Splitting the work across roles doesn’t make the system slower to be safe — it makes each agent’s boundaries explicit. The solution-architect, for instance, is tool-gated to read-only: it can propose an ADR, but it cannot write code or skip a stop. Every agent shares the same discipline — a documented refusal point, an explicit escalation path, and the rule that CLAUDE.md wins any conflict.
The pipeline: a gate on every arrow
A set of slash commands drives the work from PRD to running code — /prd:review, /prd:to-epic, /epic:decompose, /epic:estimate, /story:start, then /story:complete and /epic:done. Each one hands off to a role-shaped agent and then stops at a boundary that agent’s CLAUDE.md declares. The protocol is the same four steps every time: detect the boundary, propose two or three real options, ask the room and block — no “I’ll assume X for now” — then record the decision as an ADR before anything gets written.
Crucially, the command owns the stop, not the agent. Only the orchestrating command can put a question to the room or write a file; the role agents are advisory, reading and reasoning but never allowed to quietly write the artefact and skip the gate. That separation is what makes the discipline survive contact with a model under pressure to just finish.
There are two distinct loops, not one. Decomposition-time gates — in /prd:to-epic and /epic:decompose — catch questions that block planning: where to store events, how to map PayHub’s identities onto the local system. Implementation-time gates, inside /story:start, catch questions that are only visible once an engineer opens the story — these aren’t planning failures, they’re decisions that genuinely cannot be made before the build starts. A story’s decision trail growing after /story:start is the visible proof that the second loop fired.
Completion is verified, never self-declared: /story:complete and /epic:done require the qa-lead to map acceptance criteria and the PRD’s own success metrics to evidence, and the code-reviewer to confirm the hard architectural rules held, before a human — not an agent — marks anything done.
The toolbelt
Alongside the pipeline we walked through the supporting tools that make long agent sessions tractable: cxpak for deterministic, reviewable repo context instead of letting the model grep around with a flashlight; rtk, a Rust binary installed as a Claude Code hook that transparently compresses noisy command output before it reaches the model; and two contrasting takes on session memory — claude-mem’s automatic capture versus memory-palace’s deliberate, MCP-based remember/recall/forget. We framed the choice between them as HITL too: the discipline isn’t picking the fanciest memory tool, it’s being explicit about what the agent is allowed to assume between sessions.
What the room walked away with
By design, we did not finish the implementation — we finished the CLAUDE.md. Each participant left with a filled-in CLAUDE.md for the webhook-receiver project, a partially-built but running receiver demonstrating the hot/warm/cold split, and a working mental model for placing HITL checkpoints in their own projects. The take-home recipe is three questions against any project’s own CLAUDE.md: decide what you want the agent to assume, under an ## Architecture constraints heading; decide what you want it to ask, under ## HITL checkpoints; and decide what you want it to refuse outright. The pipeline’s commands read those headings literally, so the same gates fire on whatever boundaries a different project’s CLAUDE.md declares.
If there’s one thing worth carrying out of the room: stop the agent in code, not in prompts. A prompt that asks an agent to “check with the user first” is a suggestion the agent can rationalise past under pressure. A command that owns the block, backed by tool grants that make certain agents read-only, is a boundary. Headings are the contract, and every gate produces an ADR that outlives the meeting — the audit trail is the discipline, not a side effect of it.
Resources
- Workshop repository: github.com/cefothe/hitl-engineering-jprime-2026-workshop — public, replayable step by step
- Slides