← All posts

Our job queue is one Postgres UPDATE - so we raced it until it broke

We benchmarked the atomic claim behind Rezee's CI queue against the naive version and FOR UPDATE SKIP LOCKED - 100 concurrent workers, 10,000 jobs, real numbers on duplicates, wasted claims, latency, and the surprising way a guarded queue slows down under contention.

Apr 1, 2026 · 17 min read · Kash Gohil

Rezee is one workspace for the whole product development lifecycle - code, planning, CI/CD, chat, and docs. Its CI system has no message broker: the queue is a status column in Postgres, and the entire coordination story is one conditional UPDATE. In that post we called this design "extremely hard to break." That's the kind of sentence you should have to prove.

So we raced it. A benchmark harness (scripts/bench/queue-claim in our repo) queues 10,000 jobs and lets 1 to 100 workers fight over them in a tight loop, three different ways. Every number below is reproducible with two commands; the harness, the results JSON, and the chart generator are all committed. Numbers are from an Apple M-series laptop against Postgres 16 in Docker (tmpfs data dir), so treat them as shapes, not absolutes.

What exactly are we racing?

Three claim strategies, same queue table. The naive version reads a candidate, then updates it, with nothing in between:

SELECT id FROM jobs WHERE status = 'pending' ORDER BY id LIMIT 1;
UPDATE jobs SET status = 'running' WHERE id = $1;  -- no status check

The guarded version is what Rezee ships in makima's claim endpoint. Same read, but the write re-checks the status, so only one winner is possible:

const updated = await db
	.update(pipelineRuns)
	.set({ status: "running", startedAt: new Date() })
	.where(and(eq(pipelineRuns.id, runId), eq(pipelineRuns.status, "pending")))
	.returning({ id: pipelineRuns.id });
if (updated.length === 0) return status(409, "already claimed");

And SKIP LOCKED, the Postgres-native answer, where the database hands each worker a different row in a single statement:

UPDATE jobs SET status = 'running'
WHERE id = (
  SELECT id FROM jobs WHERE status = 'pending'
  ORDER BY id LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id;

Where does this run in the real system?

The benchmark isolates a race that lives across two services in production. kobeni (the CI runner, services/kobeni/main.go) polls and claims through makima's internal API - it never touches Postgres directly:

func (r *kobeni) poll() error {
	var resp struct {
		Runs []pendingRun `json:"runs"`
	}
	if err := r.get("/internal/pipeline/pending", &resp); err != nil {
		return err
	}
	for _, run := range resp.Runs {
		go r.execute(run)
	}
	return nil
}

func (r *kobeni) execute(run pendingRun) {
	// Atomically claim the run; another kobeni may beat us to it.
	if err := r.post("/internal/pipeline/runs/"+run.ID+"/claim", nil, nil); err != nil {
		return
	}
	...

Note what that shape implies. /pending returns up to five runs (limit(5) in services/makima/src/modules/pipelines/index.ts), and every polling kobeni instance gets the same five - the endpoint doesn't reserve anything. Reservation happens only at /claim, so with N runners the same run is routinely seen by all N and claimed by one; the other N-1 get their 409 inside execute and simply return. The read is shared, the write is exclusive, and no state exists in between - which is why the runner needs no queue library, no lease bookkeeping, and no memory of what it saw last poll. It also means the pending payload can be rich: makima attaches the full job/step tree and decrypts the repo's secrets right there at dispatch, because handing a loser a payload it never uses costs nothing.

What happens without the guard?

At two workers - not a hundred, two - 9,763 of 10,000 jobs were claimed more than once. The gap between the read and the write is a few hundred microseconds on localhost, and it's enough: both workers read the same head-of-queue row, both write, both think they won. In CI terms, both runners clone the repo and run your pipeline. Twice the compute, twice the deploys.

It compounds with concurrency, because every duplicate claim re-runs the loop and collides again:

The worst part is that the naive queue looks healthy. Its throughput counter reads ~6,000 claims/sec - the highest of any strategy - because duplicate work counts as work. Nothing errors. You find out from your cloud bill, or from a deploy that ran twice.

Here's the race, slowed down enough to watch. Toggle the guard and race four workers over one queue:

Why does one UPDATE work, mechanically?

It's worth being precise about why the guard is safe, because the answer isn't "UPDATEs are atomic" - it's a specific behavior of READ COMMITTED, the isolation level Postgres runs at by default and the one this system uses.

The naive version is the textbook lost update. Both workers' SELECTs read the same snapshot and see the same pending row. Each then issues an UPDATE whose WHERE clause is only id = $1 - a condition that's still true no matter what the other worker did. READ COMMITTED is perfectly happy to let the second UPDATE overwrite the first. No error, no conflict, nothing to retry: the anomaly is permitted, which is why the failure is silent.

The guarded version changes what happens at the moment of contention. When two UPDATEs target the same row, the second one blocks on the first one's row lock. When the winner commits, Postgres does something specific for the waiter at READ COMMITTED: it re-fetches the newly committed version of the row and re-evaluates the WHERE clause against it (the executor calls this an EvalPlanQual recheck). The re-check sees status = 'running', the condition status = 'pending' is now false, and the UPDATE matches zero rows. The guard works because the status check rides inside the locking protocol, where the naive version's check happened in a separate, earlier, unprotected read.

You can see the guard sitting exactly where it needs to be in the query plan:

Update on jobs  (actual time=0.011..0.012 rows=1)
  ->  Index Scan using jobs_pkey on jobs  (actual time=0.008..0.008 rows=1)
        Index Cond: (id = 5001)
        Filter: (status = 'pending'::text)   ← re-evaluated on the locked row
Execution Time: 0.027 ms

That Filter line is the entire safety mechanism, and it's the same line whether the row is contended or not - which is why the guard costs nothing when there's no race and degrades gracefully (409, not corruption) when there is one.

Two boring-but-load-bearing footnotes. First, this only holds because the claim is one statement: a SELECT-then-UPDATE inside a transaction at READ COMMITTED is still racy, because each statement gets its own snapshot. Second, at SERIALIZABLE the naive version would abort instead of corrupting - but then every winner pays serialization overhead to protect against a race the guard eliminates for free.

What does the guard cost?

Correctness first: zero duplicates at every concurrency level. The claim either flips pending → running or matches zero rows; there is no interleaving that produces two winners, and 700,000 tight-loop claim attempts didn't find one.

The cost is the lost races. A worker that loses gets a 409 and retries, and because every idle worker chases the same head-of-queue row, losses grow much faster than workers:

And that contention has a throughput bill. This was the genuinely surprising result: the guarded queue gets slower as you add workers.

Head-of-queue contention is a structural property of "everyone claims the oldest pending job." The guard makes the race safe; it does not make the race cheap.

So should we switch to SKIP LOCKED?

On these numbers, eventually - but not yet, and the numbers also say why not yet. SKIP LOCKED wins the torture test: no duplicates, no wasted attempts, 20x the throughput at high concurrency. Its own cost shows up in tail latency, because rows stay locked while transactions complete:

SKIP LOCKED also hides a cost that only shows up as the queue drains, and the query plan makes it visible. Its inner SELECT walks the primary key looking for a pending row - which means walking over every already-consumed row first. Halfway through a 10,000-job queue:

->  Index Scan using jobs_pkey on jobs
      Filter: (status = 'pending'::text)
      Rows Removed by Filter: 5001        ← walked over the consumed half
      Buffers: shared hit=10057
Execution Time: 2.277 ms

10,057 buffer hits to find one row. The fix is a partial index that only contains what the queue cares about:

CREATE INDEX jobs_pending_idx ON jobs (id) WHERE status = 'pending';

Same query, same table state: 14 buffers, 0.207 ms - 11x faster, ~700x less data touched, and the index shrinks as the queue drains because consumed rows fall out of it. If you run any polling queue on Postgres, this index is not optional at scale; our benchmark ran without it precisely so the strategies competed on their locking behavior rather than on index tuning, and the reproduce section below shows both configurations.

The summary table, at the two ends of the range:

Here's the production context that decides it: Rezee runs a handful of kobeni instances, and they poll every five seconds - they don't spin in a tight loop. Our real-world concurrency is the left side of every chart, where the guarded claim does 1,100+ claims/sec with sub-2ms p99 and a wasted attempt costs one cheap round trip. The 409 path isn't an error; it's the design working. We keep the guarded UPDATE because the claim crosses an HTTP boundary (runners ask the API to claim; they never touch the database), and "the loser gets a 409" is an API semantic any runner - or any agent - can understand. When runner count grows to where the wasted-attempt curve matters, the fix is moving the row selection inside the claim endpoint's query with SKIP LOCKED - a one-statement change on the server, invisible to every runner. The benchmark told us where that line is; we're nowhere near it.

One more reason the guarded shape earns its keep: it generalizes. The same conditional UPDATE is now the mechanism behind the approval gate on agent tasks - WHERE state = 'awaiting_review' instead of WHERE status = 'pending' - which means the gate inherits everything this post measured instead of introducing a second coordination mechanism to trust.

What do 700,000 status flips do to the table?

A status-column queue means every job's lifecycle is at least two UPDATEs, and in Postgres an UPDATE never modifies a row in place - MVCC writes a whole new row version and leaves the old one behind as a dead tuple for vacuum. A queue table is therefore a churn hotspot by construction, so we measured what actually happens rather than gesturing at "bloat":

• Plain table, default settings: 5,002 status updates on a 10k-row table left n_dead_tup = 0 by the time stats settled - page pruning reclaimed the dead versions opportunistically, no autovacuum required. At CI scale (thousands of runs a day, not millions an hour), churn on this table is a non-issue.
• HOT updates need headroom. A heap-only-tuple update (the cheap kind: no index maintenance) requires free space on the same page, and freshly bulk-loaded pages are packed full. Measured: at default fillfactor, 0 of 1,000 status updates were HOT; recreate the table WITH (fillfactor = 70) and 448 of 1,000 were.
• The partial index taxes every update. Here's the trap: status appears in jobs_pending_idx's predicate, so any update that changes status can no longer be HOT - measured, the fillfactor-70 table's HOT rate went from 448/1,000 back to 0/1,000 the moment the index existed. The index that makes SKIP LOCKED fast makes every claim write index entries.

So the three knobs - partial index, fillfactor, HOT rate - form a real trade-off triangle, not a checklist. For rezee's volumes we take the boring corner: no partial index, default fillfactor, and let pruning do its job. A team running six-figure daily job counts would take the opposite corner and pay the index maintenance for the scan speed. The point of measuring is knowing which corner you're in.

How do you reproduce this?

Two commands, from a checkout of the repo:

docker compose -f scripts/bench/compose.yml up -d --wait
bun scripts/bench/queue-claim/run.ts

The harness (scripts/bench/queue-claim/run.ts) gives each worker its own connection, like real runner instances; duplicates are detected from a claims audit table, not from the workers' own opinion of what they won. charts.ts regenerates every figure in this post from the results JSON - the charts are build-time SVG, no charting library, colored by the site's own CSS variables.

What doesn't this prove?

Benchmarks mislead by omission, so, plainly:

• It's a torture test, not a simulation. Workers claim in a tight loop with zero think time. Production kobeni polls every 5 seconds; its real contention is close to zero. The torture test finds the failure shapes, not the operating point.
• Localhost flatters everything. Sub-millisecond round trips widen or narrow every window. Real network latency makes the naive race worse (bigger read-to-write gap) and makes wasted attempts pricier.
• The guarded p99 only counts wins. A worker may lose several races before a win; the "time to get a job" distribution is fatter than the per-claim one. Wasted attempts are reported separately for exactly that reason.
• One table, one queue, no job payloads. Vacuum pressure from high-churn status updates on a busy production table is its own topic.
• A claim is not a completion. If a runner dies after claiming, the job stays running forever. That's the missing lease heartbeat we called out in the kobeni post, and this benchmark doesn't touch it.

FAQ

Why not just use SELECT FOR UPDATE without SKIP LOCKED?

Plain FOR UPDATE makes every other worker queue behind the row lock, so claiming serializes: you get correctness with the worst of both worlds - waiting and no parallelism. SKIP LOCKED exists precisely so workers skip contested rows instead of lining up behind them.

Why not use a real message broker for CI jobs?

Because the control plane already has Postgres, and a status column gives you a queryable queue, transactional state changes with the rest of the run's data, and crash-consistent recovery for free. The benchmark puts numbers on the ceiling: even the contended guarded claim sustains hundreds of claims per second, and team-scale CI needs a few per minute. A broker earns its operational cost somewhere past that ceiling, not before.

Does 409-on-conflict mean runners hammer the API with retries?

No - a 409 sends the runner back to its normal poll loop, not into a hot retry. At production polling intervals, conflicts are rare (two runners must ask within the same few milliseconds); the benchmark's 22-losses-per-win figure is what a 100-worker tight loop looks like, which is the point of a torture test.