The life of a CI log line (or - how "live" do live logs need to be?)

How a log line travels from a build container to your browser in Rezee - scanner, redaction, 20-line batches, one Postgres row per line, and 2-second polling - and why we chose boring over push.

May 29, 2026 · 4 min read · Kash Gohil

When your pipeline runs in Rezee, log lines appear in the browser with a blinking cursor, close enough to live that nobody asks questions. This post follows one line through the whole pipe - and makes the unfashionable argument that got it built this way: for log content, there is no WebSocket, no server-sent events, no stream. There are batches, rows, and a 2-second poll. (Run status has since moved to the workspace SSE stream - and that post explains why log content deliberately stayed on the design below.) Here's the path, and why.

Step 1: container to runner

Inside kobeni, our CI runner, the step's container runs with stderr merged into stdout - build tools disagree philosophically about which stream progress belongs on, and users don't care - and a bufio.Scanner reads the pipe line by line.

Before a line goes anywhere, it's redacted. Every secret value known to the run is literal-replaced with ***, in the runner, before transmission - so a leaked token never exists downstream at all:

func redact(line string, secretValues []string) string {
	for _, v := range secretValues {
		if v != "" {
			line = strings.ReplaceAll(line, v, "***")
		}
	}
	return line
}

Step 2: runner to API, in batches

Lines accumulate into a batch; at 20 lines - or at end of stream - the batch flushes as a single POST to the API:

scanner := bufio.NewScanner(rc)
var batch []string
flush := func() {
	if len(batch) == 0 {
		return
	}
	body := map[string]any{"lines": batch}
	r.post("/internal/pipeline/steps/"+stepID+"/logs", body, nil)
	batch = batch[:0]
}
for scanner.Scan() {
	batch = append(batch, redact(scanner.Text(), secretValues))
	if len(batch) >= 20 {
		flush()
	}
}
flush()

Twenty is a compromise number: chatty enough that a healthy build's output appears in near-real-time, batched enough that a make spewing thousands of lines doesn't turn into thousands of HTTP requests.

Step 3: API to Postgres, a row per line

The API bulk-inserts each line as its own row: {stepId, line, createdAt}. One row per log line sounds extravagant until you price it: a large run is tens of thousands of rows, which is nothing, and in exchange logs are queryable, retained, and served with the same database access path as everything else. No log files to rotate, no blob storage to page against, no second system.

Step 4: Postgres to browser, by polling

The run page fetches the run's log rows and re-fetches every 2 seconds while the run is pending or running - TanStack Query's refetchInterval, nothing more. The blinking ▋ cursor on a running step is cosmetic. Honesty in one sentence: "live logs" means at most two seconds stale, delivered by the most boring mechanism on the web.

Why not a real stream?

Because every push technology - WebSockets, SSE - buys latency you may not need with operational weight you definitely pay: connection state on the server, reconnection and backfill logic on the client, load-balancer and proxy configuration, and a second delivery path to debug when it half-works. Polling through the query layer we already had cost roughly zero marginal code, keeps the API stateless, and degrades gracefully (a missed poll is just the next poll).

The honest measure for CI logs: a human watching a build reacts on the scale of seconds, and the runner batches at 20 lines anyway - sub-second browser delivery would be precision faked at the last hop. This is the same polling-until-push-is-earned stance the rest of the product takes: choose boring, measure, upgrade the specific surface when the cost is real.

What "earned" will look like here, eventually: a sequence number on log rows (ordering currently leans on createdAt timestamps), offset-based incremental fetches instead of refetching the run's full log set, and possibly SSE for the final hop - each a measured response to scale, not an architecture statement.

FAQ

How live are Rezee's CI logs exactly?

Worst case a line waits for its 20-line batch (or end of step) plus up to one 2-second poll - in practice a healthy build's output appears within a couple of seconds of being printed. Failure output lands immediately, since end-of-stream forces a flush.

The shape has since outgrown CI: agent task streams - the step-by-step trail a human watches while an agent works - use the identical pattern, append-only rows plus a keyset poll, because "rendered output can never change behind you" turns out to be exactly what you want for an audit trail too.

Why store logs in Postgres instead of files or object storage?

Team-scale CI log volume is tiny by database standards, and rows buy search, retention policy, and a single access path with the auth the API already enforces. Object storage becomes right at a scale where a migration behind the same endpoint is straightforward.

Isn't polling wasteful?

It's bounded and cheap: one indexed query per open run page every 2 seconds, only while a run is active. The comparison isn't polling versus free - it's polling versus holding, load-balancing, and debugging persistent connections. At current scale, the wasteful choice would be the stream.