← All posts

Production latency was round trips, not queries - what 2.4 seconds of cold launch was actually made of

Every query returned in 1 to 5 milliseconds and the box was idle. The slow endpoints were serial awaits, and the worst one was justified by a code comment that was confidently wrong.

Sep 22, 2026 · 10 min read · Kash Gohil · LinkedIn · X

Rezee's production database is small. Tens of documents, hundreds of issues, a box that sits near idle. So when the sync bootstrap took 2.4 seconds and a document page took 85 milliseconds at p50, the instinct was to look for a slow query.

There were none. Every query we measured returned in 1 to 5 milliseconds. A round trip from the API to Postgres costs roughly 2 to 5 milliseconds. Latency was not query cost and it was not load. It was query count in series, and the arithmetic held everywhere we checked it:

  • about 15 serial reads on GET /docs/:number, about 85ms p50
  • about 28 serial collection reads in the sync bootstrap, about 880ms p50 for that portion

Multiply the number of awaits by a few milliseconds and you have the endpoint's latency. That is the whole finding. This post is how the two worst offenders got that way, because both of them were written by somebody being careful rather than somebody being careless.

Shape one: an object literal is a sequence

GET /docs/:number built its response inline:

return {
  ...rest,
  author: await authorName(doc.authorId),
  parentId: await parentNumber(doc.parentId),
  project: await projectRef(projectId),
  system: await systemNumber(systemDocId),
  branchOf: await parentNumber(branchOf),
  reviews: await reviewsForDoc(doc.id),
  issues: await linksForDoc(doc, g.issuePrefix, user?.id ?? ""),
  issueCount: (
    await db.select({ n: sql<number>`count(*)::int` })
      .from(docIssues).where(eq(docIssues.docId, doc.id))
  )[0]?.n,
  // ...
};

Eight reads. None of them reads anything another one writes. They look concurrent because they are siblings in a literal, and they are not: an object literal evaluates its properties in order, so those eight awaits are eight round trips deep rather than one wide.

The fix is the boring one, and the comment is the part worth keeping:

// Eight independent reads, and an object literal evaluates its
// properties in order - so written inline they were eight round trips
// deep rather than one wide (ISSUE-924). Nothing here reads anything
// another one writes.
const [author, parent, project, system, base, reviews, issues, linked] =
  await Promise.all([
    authorName(doc.authorId),
    parentNumber(doc.parentId),
    projectRef(projectId),
    systemNumber(systemDocId),
    parentNumber(branchOf),
    reviewsForDoc(doc.id),
    linksForDoc(doc, g.issuePrefix, user?.id ?? ""),
    db.select({ n: sql<number>`count(*)::int` })
      .from(docIssues).where(eq(docIssues.docId, doc.id)),
  ]);

Twenty-six lines changed, thirteen removed. The last sentence of that comment - nothing here reads anything another one writes - is the precondition, and writing it down is what stops the next person adding a ninth property that does.

This shape is easy to miss in review because the diff that introduces it is always small. Somebody adds one more field to a response, with one more await in front of it, and the endpoint gets 3ms slower. Do that eight times over a year and nobody can point at the commit that made the page slow.

Shape two: a comment that was confidently wrong

The sync bootstrap loops over the models a client asked for and reads each one's rows. It did that sequentially, on purpose, with a reason attached:

/**
 * Sequential rather than `Promise.all`: a transaction is one connection, so
 * there is no round trip to save, and inside a snapshot the order these are
 * read in cannot change the answer.
 */

The second clause is true. The first is false, and it is false in a way that survived review precisely because it sounds like something an experienced person would say.

postgres.js pipelines queries on a connection, including inside a transaction. Queries issued before earlier ones have returned go out back to back, and the waits overlap. "It is one connection so there is no round trip to save" describes a driver that waits for each reply before writing the next query, which this one does not do.

Sequential queries against pipelined queries on one connection sequential await, then ask ... x30 pipelined ask, then await one wait wall clock
Thirty independent reads on one connection: 2.3ms issued sequentially, 0.5ms issued together, measured on loopback. The connection was never the constraint; waiting for each reply before writing the next query was.

Thirty reads took 2.3ms sequential against 0.5ms concurrent on loopback. Over a real network hop that gap widens with every millisecond of round trip, and in production this loop was the larger half of a 2.4 second bootstrap.

The rewrite keeps the ordering-is-irrelevant half of the old comment and corrects the rest:

/**
 * Issued together rather than one after another. This used to be a sequential
 * loop on the grounds that a transaction is one connection and so there was no
 * round trip to save - which is wrong: postgres.js pipelines what is in flight
 * on a connection, so thirty reads cost one trip's wait rather than thirty.
 * Inside a snapshot the order these are read in still cannot change the answer.
 */

One detail the rewrite needed: six of the models want the reader's audience set, and none of them should ask for it twice. The fix is to share the promise rather than the result.

let audiences: Promise<Set<string>> | null = null;
const held = () => (audiences ??= audiencesFor(exec, ws.id, userId));

Storing the promise before it is awaited is what makes six concurrent callers share one query. Storing the resolved value instead - the cache shape most people reach for - means six callers all find the cache empty and all start their own.

The client half: waves, and what makes two requests safe

Fixing the server left the cold launch still asking for all thirty-odd models in one request, so the screen in front of the reader waited on every other screen's rows. Opening Docs waited on chat, pulls and runs.

So a cold bootstrap now goes in two waves: the models the opening screen reads, then everything else. The first wave lands, the surface paints, the rest arrives behind it.

Splitting one consistent read into two is the kind of optimization that quietly introduces a correctness bug, and the guard against it is worth more than the speedup:

**What makes two requests safe is the reach epoch.** Each dump says who the
reader was when it was read. If that moved between the waves - a project
closed, a grant revoked - the two halves are answers to different questions
and stitching them together would leave rows the reader may no longer see.

If the epochs differ, the client falls back to the single-request path, which was always the repair for reach moving underneath a read. And the stored cursor is the older of the two waves' cursors, so the next delta re-reads the stretch between them rather than stepping over it. Re-applying a few actions is free; skipping one is a row that stays wrong until something forces a bootstrap.

Only the cold path splits. Repairs stay one request on purpose: a store being repaired wants one snapshot, not two that can disagree.

How to find these

Two tools, in this order.

A route histogram tells you where to look. Prometheus on the box, makima_http_request_duration_seconds_bucket labelled by route, gives a p50 and p99 per endpoint. That is the list of candidates and nothing more.

Counting awaits tells you why. Before opening a query plan, read the handler and count the awaits on the path. If the count times a few milliseconds is roughly the p50, you have your answer and EXPLAIN will not add anything. On this system that test has been right every time.

The trap is that profiling the database confirms the database is fine, which feels like ruling something out and is actually a dead end. Fifteen queries at 3ms each is a healthy database and a slow page.

Where not to do this

Promise.all is not free to apply, and three cases should stay sequential.

Anything with an ordering constraint. The offline outbox replays writes strictly in the order somebody made them, because a patch that follows a create has to land after it. Parallelising that is a race whose loser is whichever edit the person made second.

Anything where one read informs another. Resolve the workspace, then read inside it. That is a genuine dependency, and the two round trips are the cost of the check.

Authorization before work. A gate that has to hold before anything else runs is sequential by definition. Issuing it alongside the work it guards means the work has already happened by the time the gate answers.

The good candidates all look the same: independent reads, of the same snapshot, whose results are assembled into one response. That was eight properties of a document and thirty collection reads in a bootstrap.

What it cost

  • Concurrency is not free to reason about. Every Promise.all is a claim that its members are independent, and that claim needs a comment, because the next person to add a line inside it will not re-derive it.
  • The shared-promise idiom is subtle. x ??= f() on a promise is correct and x ??= await f() is a thundering herd. It reads almost the same.
  • Splitting a snapshot needs a guard. The two-wave bootstrap is a genuinely more complex code path than one request, and most of its added lines are the epoch check and the cursor arithmetic rather than the speedup.
  • We fixed a comment, not just a loop. The sequential bootstrap was not an oversight; it was a deliberate decision with a wrong premise, and it survived because the premise sounded authoritative. That is the more expensive class of bug, and the only defence is measuring the premise rather than the code.

None of this was a database problem, and none of it needed an index. It needed somebody to count.

FAQ

Does Promise.all on one Postgres connection actually help?

With postgres.js, yes. It pipelines queries that are in flight on a connection, so independent queries issued together overlap their waits instead of serialising them. We measured thirty reads at 2.3ms sequential against 0.5ms concurrent on loopback. Drivers differ, so measure yours rather than assuming either way.

Why not just add more connections?

Because the connection was never the constraint. The box was idle and every query returned in single-digit milliseconds. More connections help when queries are queuing for one; they do nothing when the code is waiting for a reply it already asked for before asking the next question.

How do you tell a round-trip problem from a slow query?

Multiply the number of awaits on the request path by your round-trip time. If that lands near the endpoint's p50, the queries are not the problem. If it does not, open the query plan. The count is faster to get than the plan, so do it first.

Is 2.4 seconds of bootstrap a lot?

For a cold launch that then serves the whole workspace locally, it is tolerable; for the first paint of the screen somebody opened, it is not. That distinction is why the fix had two halves: issue the reads together so the whole dump is faster, and split the cold launch so the opening screen does not wait on the rest of it. More on what that dump contains in one store, several writers.

Does an ORM cause this?

Not really. Drizzle emits the query you asked for; the serialisation here was in how the calls were arranged in TypeScript, not in what SQL came out. The one thing an ORM does contribute is making each individual read cheap to write, which is what lets fifteen of them accumulate in a handler without anybody noticing.