← All posts

Keep the LLM loop out of the API - the beam agent runtime

Why Rezee runs agents in a separate worker service instead of inside the API - a measured 14x tail-latency blowup when the agent loop shares the request path, and the kobeni-shaped runtime that avoids it.

May 14, 2026 · 8 min read · Kash Gohil

Rezee is one workspace for the whole product development lifecycle - code, planning, CI/CD, chat, and docs - where agents are first-class teammates. An agent doing real work runs a loop: call the model, get a tool call, execute it, feed the result back, repeat, for seconds to minutes. The tempting place to run that loop is inside the API request that triggered it. This post is the measured case for why we don't - the agent runtime, beam, is a separate worker service, the same shape as our CI runner kobeni - and what it costs the whole API when you get this wrong.

What's wrong with running the loop in the request?

The obvious design: POST /agents/:id/tasks starts the agent, awaits the loop, returns the result. It's simple, and it quietly poisons your entire API. An agent loop isn't just slow - it does real synchronous CPU work between the awaits: parsing a large tool result, assembling the next prompt, counting tokens. On a single-threaded runtime (Bun, Node), that CPU work runs on the same event loop that serves every other request. One agent thinking makes every health check, every page load, every unrelated write wait its turn.

We measured it. The harness (scripts/bench/agent-runtime) runs an API with a trivial /health endpoint and an /agent endpoint, in two shapes: the loop in the request handler, and the loop in a worker thread (beam's shape). Both do the identical agent work - 8 turns, each a 40ms model wait plus an 8ms CPU burst. We hold 10 agent tasks in flight and probe /health to see how the hot path holds up.

With the loop in the API process, /health p99 goes from a 1.9ms idle baseline to 27.6ms - a 14x blowup - and the worst probe took 62ms. With the loop in a worker, p99 under the same load is 5.5ms. The API barely notices ten agents are working, because their CPU bursts are on another thread.

The latency chart undersells it, though. Look at how many requests the hot path could even serve during the load window:

The in-process API served 100 health probes while agents ran, against 189 idle - it lost nearly half its throughput to the agent CPU work stealing the event loop. The worker-model API served 187, essentially its full idle rate. This is the real cost: not just that agent-triggering requests are slow (they're supposed to be), but that unrelated requests get dragged down with them.

Why is beam shaped like the CI runner?

Because we already solved this problem once, for CI. kobeni runs untrusted, long-running, resource-heavy build steps - exactly the profile of an agent loop - and it runs them in a separate service that claims work from a Postgres queue. beam is the same pattern pointed at a different table:

  makima (API)                          beam (agent runtime)
  ────────────                          ────────────────────
  POST /agents/:id/tasks                loop:
    → INSERT agent_tasks (running)  ┌─→   claim a running task (conditional UPDATE)
    → return 202 immediately        │     run the LLM/tool loop:
                                    │       call model
  the request is DONE here ─────────┘       execute tool via makima's API
                                            append agent_task_steps
                                          until done or awaiting_review

The API's entire job is one INSERT and a 202. It never holds the loop, never runs model-response parsing, never blocks on a tool call. Everything expensive happens in beam, which can crash, restart, or scale independently without touching the API's latency. The claim is the same conditional UPDATE kobeni uses - so multiple beam workers coordinate with zero new machinery - and the loop drives the agent_tasks state machine we detailed separately.

What else does the separation buy?

Latency is the measurable headline, but three architectural properties matter as much and are why "just use a worker thread inside the API" isn't the real answer either:

• Deploys don't kill work. Ship a new API version and every in-flight request dies with the old process. If those requests are agent tasks, a routine deploy murders every agent mid-thought. With beam separate, you deploy the API freely; beam drains its own tasks on its own schedule. This is the same reason CI runs survive an API deploy.
• Blast radius is bounded. An agent loop that wedges - a runaway tool call, a model that won't stop - takes down a beam worker, not the API. A wedged in-process loop takes a chunk of your API's event loop with it. The failure domain of "an agent misbehaved" should not include "nobody can load the issue board."
• Independent scaling. Agent load and API load are uncorrelated - a quiet workspace with three agents churning through a migration, or a busy workspace with none. Separate services scale on separate signals. You add beam workers when the agent_tasks queue backs up, and API instances when request latency climbs, and neither decision drags the other.

The worker-thread version buys you the latency win (the CPU is off the request event loop) but none of these three - the thread still dies with the API process on deploy, still shares its memory and crash domain, still scales as one unit. Threads fix the symptom this benchmark measures; a separate service fixes the thing the benchmark can't.

What's the honest cost of this design?

A separate service is not free, and pretending otherwise would fail our own depth bar:

• A tool call is now a network hop. When beam executes a tool - comment on an issue, open a PR - it calls makima's API over HTTP with a scoped agent token, rather than calling an in-process function. That's real latency per tool call and a real auth surface. We think it's the right trade (the tool surface is the same one external agents use over MCP, so it has to be a real API anyway) but it is a cost.
• The task is now asynchronous to its trigger. POST /tasks returning 202 means the caller can't get the result inline - it watches the task stream instead. That's more moving parts than "await the answer," justified only because agent work is genuinely long.
• Two services to operate. beam has its own deploy, its own health, its own scaling. For a small team that's real overhead - the same overhead kobeni already imposes, which is partly why the pattern is worth reusing rather than inventing twice.

FAQ

Isn't an async LLM call already non-blocking? Why would it hurt the API?

The await on the model call is non-blocking - that part yields the event loop fine. The damage is the synchronous work between awaits: parsing a multi-kilobyte tool result, building the next prompt, token accounting. That runs to completion on the event loop and blocks everything else while it does. Our benchmark models it as 8ms of CPU per turn, which is conservative for a loop handling real tool outputs. A pure-async loop with zero CPU work wouldn't show this - but no real agent loop is pure-async.

Why not just a worker thread pool inside the API instead of a whole service?

A thread pool fixes the latency (the CPU leaves the request thread) but not deploy safety, crash isolation, or independent scaling - the loop still lives and dies with the API process. Since we already run kobeni as a separate worker service and the pattern is proven, extending it to beam is less new machinery than a bespoke in-API thread pool, and it gets all three properties threads don't.

How does beam coordinate if you run several of them?

The same way several kobeni instances coordinate: each claims a task with a conditional UPDATE (WHERE state = 'running' AND ...), the loser gets a no-op and moves on, and there's no broker or lock service. We benchmarked that claim to 700,000 contended attempts with zero double-claims, so beam inherits a coordination story we've already tested rather than a new one.