← All posts

Catching flaky tests from results - and quarantine with an owner and an expiry

Rezee parses a job's JUnit reports into test results, counts a flake when one commit sees a test both pass and fail, and lets a team quarantine a flaky test only with an owner and an expiry. The exact rule, the lock that keeps it correct, and the runner change quarantine needed.

Oct 6, 2026 · 15 min read · Kash Gohil · LinkedIn · X

A flaky test teaches a team to click re-run. After a month of that, a red pipeline means "probably fine", and the pipeline has stopped refereeing anything. Everybody agrees flaky tests are bad. The harder question is a narrow one: what evidence lets software call a test flaky, as opposed to broken, slow, or failing for a reason nobody has found yet?

Rezee's answer is one rule. A test is flaky when one commit has seen it both pass and fail. This post explains that rule and why we chose it over a pass-rate threshold. It covers the lock without which the rule misses flakes, what the rule cannot see, and quarantine. Quarantine took a second commit to the runner before it worked the way the first commit claimed.

A report is an artifact

Every test runner can write JUnit XML, so that is the one format we read. A job names its reports with test_report: in .rezee/actions.yml:

jobs:
  test:
    image: golang:1
    steps:
      - name: test
        run: gotestsum --junitfile reports/unit.xml ./...
    test_report: reports/*.xml

The globs need nothing new from the runner. When makima reads the pipeline file, it adds the report globs to the job's artifacts. The runner uploads them the way it uploads any artifact. Makima reads the archive when the artifact is registered. The first slice shipped without a new runner image, and that was the point of doing it this way.

The parser is about 180 lines of our own, not an XML dependency. A JUnit report is elements, attributes, text and CDATA. Nothing in it needs namespaces or DTDs. Each <testcase> becomes a case. Its suite is its classname, which is where pytest, Go and JUnit itself say the test lives. If there is no classname, the enclosing <testsuite> name is used. If a case has more than one outcome element, a failure outranks an error, and an error outranks a skip.

Two tables hold the result. test_cases is a test's identity across runs. test_results is one outcome on one job:

CREATE TABLE IF NOT EXISTS test_cases (
	id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
	repo_id uuid NOT NULL REFERENCES repositories (id) ON DELETE CASCADE,
	suite text NOT NULL,
	name text NOT NULL,
	created_at timestamptz NOT NULL DEFAULT now(),
	last_seen_at timestamptz NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX IF NOT EXISTS test_cases_repo_suite_name_idx
	ON test_cases (repo_id, suite, name);

CREATE TABLE IF NOT EXISTS test_results (
	id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
	run_id uuid NOT NULL REFERENCES pipeline_runs (id) ON DELETE CASCADE,
	job_id uuid NOT NULL REFERENCES pipeline_jobs (id) ON DELETE CASCADE,
	case_id uuid NOT NULL REFERENCES test_cases (id) ON DELETE CASCADE,
	status test_result_status NOT NULL,
	duration_ms integer NOT NULL DEFAULT 0,
	message text,
	detail text,
	created_at timestamptz NOT NULL DEFAULT now()
);

Identity is (repo, suite, name). Rename a test and it is a new test with no history. That is the honest behaviour, because nothing in a JUnit file says that two names are the same test.

There are limits on what one upload can cost. An archive larger than 100 MB is skipped. Inside it, an XML file larger than 50 MB is skipped. A report stops at 50,000 cases. A failure message is clipped at 1,000 characters and a stack at 16,000.

The first wrong turn: the run finished first

The first version read the report in the background. The artifact upload answered at once, and parsing happened after. That looked harmless, and it was wrong for the page that mattered most.

A job uploads its artifacts at the end. The run can finish moments later. Somebody opens the red run, and the failed tests are not there yet, because the parse is still going. The next commit made the upload wait for the parse:

try {
	const report = await ingestTestReport(artifact);
	if (report.written > 0) await emitRun(artifact.runId);
	...

The rows now exist before the run can finish. A red run leads with the tests that failed, with their messages and stacks, one click from the log. Ship's runs list names the failing test in place of the step. "TestRefusesJobWithoutKey failed" is more useful than "step test failed", which describes almost every red run.

What counts as a flake

This is the whole rule, from the ingest transaction:

// A flake is a commit that has now seen a test both pass and fail, and
// was not already counted as one (ISSUE-981).
const now = new Map<string, Outcome>();
for (const c of cases) {
	const id = ids.get(keyOf(c.suite, c.name)) as string;
	const seen = now.get(id) ?? { ...(before.get(id) ?? NEITHER) };
	if (c.status === "passed") seen.passed = true;
	if (c.status === "failed" || c.status === "error") seen.failed = true;
	now.set(id, seen);
}
const flaked = [...now]
	.filter(([id, seen]) => {
		const was = before.get(id);
		return seen.passed && seen.failed && !(was?.passed && was.failed);
	})
	.map(([id]) => id);

Read precisely, it says the following:

1. The unit is one test in one repository at one commit sha. Every run of that sha counts, in any job and any pipeline run.
2. A result of passed marks the commit as having seen the test pass. A result of failed or error marks it as having seen the test fail. skipped marks neither.
3. before is what the database already held for that commit. It is computed with bool_or over every earlier result for the same repository and sha.
4. A test flakes when this report moves its commit from "not both" to "both". Then test_cases.flakes increases by one and flaked_at becomes now.
5. A commit that already had both is not counted again. Five re-runs of one commit are one flake, not five.

A test is called flaky in the UI if flaked_at is within the last thirty days.

Why the commit? Because a re-run builds the same sha. The commit message for this slice says it in one line: same sha covers re-runs, so no re-run provenance is needed. We never record which run repeated which, or whether a person or a retry policy pressed the button. The code did not change between the two results. The test did something different. That is the evidence.

A worked example. In the runner repository, sandbox.TestRefusesJobWithoutKey fails on commit 4be1c09. kash re-runs the pipeline and it passes:

The test now shows as flaky on the run page and on Ship. On a green re-run, the run page names what flaked on that commit. It shows the failure that the re-run was for, which otherwise vanishes the moment the run turns green.

The lock, or how two jobs miss a flake

The rule reads before and then writes. That is a race the first time two jobs on one commit finish together.

Say the run has a unit job and an integration job, and both run the same test. One fails it and one passes it. Both reports arrive in the same second. Each transaction reads before, and neither sees the other's rows, because neither has committed. Each one alone sees only one side. Neither counts the flake, and the evidence is in the database, uncounted.

So each ingest takes a transaction-scoped advisory lock on the commit first:

// One commit's reports at a time, so two jobs finishing together cannot
// each miss the other's half of a flake.
await tx.execute(
	sql`select pg_advisory_xact_lock(hashtext(${`tests:${job.repoId}:${job.sha}`}))`,
);

Reports for different commits never wait on each other. Reports for one commit go one at a time, and the second always sees the first. hashtext can collide, and a collision makes two unrelated commits wait for each other briefly. It costs a little latency and never a wrong count. We have written about the same family of locks for stacked pull requests.

What the rule cannot see

We chose a rule that counts only direct evidence. We accepted that it does not count everything, and that the evidence can mislead in one case. These are the gaps, in the order we expect people to hit them.

A flake on a commit that nobody re-runs leaves no evidence. It fails once, somebody pushes a new commit, and the new commit passes. The history shows that as broke and fixed, not as a flake. A team that never re-runs gets few flakes counted. We think that is the right failure. The alternative is to guess from pass rates.

Two jobs that run the same test under different conditions look like a flake. Identity is repository, suite and name. Say one job runs a suite on one database version and another job runs it on a second. A test that fails only on the second reads as one test that passed and failed on one commit. If you run the same suite twice with different conditions, a distinct suite name in the report keeps them apart.

A runner that retries in place and writes both attempts into one report counts as a flake. That is correct. It surprises people who think of the retry as invisible.

Why not a pass rate? A test with a 90% pass rate over thirty days might be flaky. It might also have been broken for three of those days and fixed. The pass rate cannot tell those apart, and the second case is not a flake. Same commit, different outcome can.

A test's thirty days

Each test has a page. It shows pass rate and median duration over thirty days, recent results, and the commits where its behaviour changed.

Pass rate is passed over passed plus failed. Errors count as failures. Skips are left out, because a skipped test said nothing about the code. The median comes from percentile_disc(0.5) over durations that are not skips. A skip takes zero milliseconds and would pull the median down.

The changes list is the part people use. Results are read oldest first and folded by commit. Each commit is a pass, a fail, or flaky if it saw both. Then each commit is compared with the last commit that settled anything:

const kind =
	commit.outcome === "flaky"
		? "flaked"
		: before === "pass" && commit.outcome === "fail"
			? "broke"
			: before && before !== "pass" && commit.outcome === "pass"
				? "fixed"
				: null;
...
// A flaky commit settles nothing, so the next one is read against the
// last commit that did.
if (commit.outcome !== "flaky") before = commit.outcome;

That last line is the decision. Without it, a pass after a flaky commit would read as "fixed", and the list would claim a fix that never happened. The page reads the latest 200 results and lists up to 50 changes, each linked to the run that showed it.

The slowest suites, and what they cost

Ship's Tests tab answers a different question: where the minutes go. For each suite, it sums durations per job, takes the median of those sums as time per run, and adds them all for the cost over thirty days:

with per_job as (
	select c.repo_id, c.suite, r.job_id,
		sum(r.duration_ms) as ms, count(distinct r.case_id) as tests
	from test_results r
	join test_cases c on c.id = r.case_id
	where c.repo_id in (...) and r.created_at >= now() - make_interval(days => 30)
	group by c.repo_id, c.suite, r.job_id
)
select repo_id, suite, max(tests)::int as tests, count(*)::int as runs,
	percentile_cont(0.5) within group (order by ms) as per_run_ms,
	sum(ms)::bigint as total_ms
from per_job
group by repo_id, suite
order by total_ms desc
limit 10

Time per run is the annoyance. Total minutes is the bill. They sit side by side because they are the same line. Sorting by the total puts a fast suite that runs on every push above a slow one that runs nightly. The tab lists the ten costliest suites, the fifteen costliest tests, and the flaky ones. It shows minutes, not money. A rate depends on your runners.

Quarantine, with an owner and an expiry

Most quarantine systems are a list of test names in a file. A name goes in during an incident and nobody takes it out. A year later the list is a second, hidden test suite that nobody runs.

We kept the mechanism and changed who is responsible for it:

CREATE TABLE IF NOT EXISTS test_quarantines (
	id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
	case_id uuid NOT NULL REFERENCES test_cases (id) ON DELETE CASCADE,
	owner_id uuid REFERENCES users (id) ON DELETE SET NULL,
	reason text NOT NULL DEFAULT '',
	created_by uuid REFERENCES users (id) ON DELETE SET NULL,
	created_at timestamptz NOT NULL DEFAULT now(),
	expires_at timestamptz NOT NULL,
	lifted_at timestamptz,
	lifted_by uuid REFERENCES users (id) ON DELETE SET NULL
);
-- One live quarantine per test.
CREATE UNIQUE INDEX IF NOT EXISTS test_quarantines_live_idx
	ON test_quarantines (case_id) WHERE lifted_at IS NULL;

Four rules sit on top of that table.

Only a flaky test can be quarantined. The function checks flakes first and refuses with a sentence: "only a flaky test can be quarantined; this one has never passed and failed on one commit". A test that fails every time is broken. Hiding it is how a suite dies.

Every quarantine has an owner and an expiry. The owner must be a member of the workspace. If you name nobody, the owner is you. The expiry is 1 to 90 days and defaults to 30. Quarantining a test that is already quarantined changes its owner, reason or expiry. It does not stack a second row.

An expired quarantine stays in force. This was the decision we argued about most. Lifting it automatically would turn unrelated pull requests red on a random morning, for a failure nobody on them caused. Keeping it silently would rebuild the forgotten list. So it stays, and it is surfaced: Ship's Tests tab lists quarantines with the expired ones first, flagged, with the owner's name beside each. Lifting is a person's call, never the clock's.

A quarantined test still runs and still records. Its results go into test_results like any other, with one extra column, quarantined, written at the time. If the quarantine is lifted next week, the history still says which failures were hidden when they happened.

Then the job. A job passes by quarantine only when all three of these hold:

if (!facts?.stepFailed) return false;
const cleared = facts.failing > 0 && facts.unquarantined === 0;
const want = cleared ? "success" : "failure";

A step failed. At least one test in the job's reports failed. Every failing test was quarantined. A failure that no report explains stays a failure. A job that crashed before writing a report, or failed for a reason outside the tests, is never passed by this.

The second wrong turn: the runner had already decided

The first quarantine commit did all of that in makima. When a report arrived, makima moved a failed job to passed if quarantine cleared it. The run page showed the right answer.

But the runner had already reported the job as failed before it uploaded the artifacts. In Rezee, a failed job skips every job that needs: it. By the time makima corrected the status, a deploy job downstream could already have been skipped. The status was right and the pipeline was not.

The fix was in kobeni, the runner. It now uploads artifacts before it reports the job. Makima's reply to the upload says whether every failing test was quarantined. If so, kobeni writes one line into the failed step's log and reports the job as passed:

cleared, err := r.uploadArtifacts(j.ID, j.Artifacts, workdir)
...
if jobStatus == "failure" && failedStep != "" && cleared {
	r.streamLogs(failedStep, strings.NewReader(quarantineNotice), secretValues)
	jobStatus = "success"
}
...
r.patchStatus("/internal/pipeline/jobs/"+j.ID+"/status", jobStatus)

kobeni: every failing test here is quarantined; the job passes

The line in the log matters. Somebody who opens a green job and sees a failed step deserves the sentence that explains it. And a makima that predates quarantine does not send cleared, which reads as false. An old API with a new runner fails jobs the old way, not a new way.

So report ingest needed no new runner image, and quarantine did. More on the runner itself in building a CI runner in Go.

The issue the fix is planned in

A quarantine with an owner is a promise, and promises in Rezee live in issues. So a test can name the issue its fix is planned in. The test's page has a button that files that issue through the ordinary create, with every rule a hand-filed issue meets. It asks which project to file it in, and offers the projects the test's repository belongs to first, so the issue lands where somebody plans that code. The link goes both ways. The test shows its issue, and the issue lists its tests, with which of them are flaky or quarantined.

The column is one nullable issue_id on test_cases. The migration's comment says what the link is for: "the fix, planned rather than remembered".

What it cost

• The rule undercounts on purpose. A team that never re-runs will see few flakes. We would rather undercount with evidence than overcount from a percentage.
• Identity is a name. A renamed test starts over. Two jobs that run one suite under different conditions share one identity unless the report says otherwise.
• Ingest is on the upload's path now. Waiting for the parse makes a large report slower to upload. That is the cost of a run page that is right the moment the run ends.
• Quarantine crossed a process boundary. The correct status needed the runner to ask makima before it reported. The runner and the API now have to agree on the order of two calls, and the runner image ships separately from the API.

FAQ

How does Rezee decide a test is flaky?

When one commit has seen the test both pass and fail. Every result for the same repository and commit sha counts, across jobs and re-runs. failed and error count as a fail. skipped counts as neither. Each commit counts as at most one flake.

Does Rezee need to know which run was a re-run?

No. A re-run builds the same commit, so comparing results by commit sha covers re-runs without recording which run repeated which.

Can I quarantine a test that always fails?

No. Only a test that has flaked can be quarantined. A test that fails every time is broken, and the right place for it is a fix or a deletion.

What happens when a quarantine expires?

It stays in force, and Ship's Tests tab lists it as expired with its owner. Rezee never lifts a quarantine on its own, because that would turn unrelated pull requests red without warning.

Does a quarantined test still run?

Yes. It runs and records like any other test. Each result is marked as written under quarantine, and a job passes only if every failing test in it was quarantined and a report explains the failure.