← All posts

Closes ACME-12 - parsing closing keywords without closing the wrong issue

How Rezee links a merged PR to the issue it closes by parsing closing keywords - tested against 500 real OSS pull requests, where the naive regex would have auto-closed the wrong issue from a code block, and the two corrections that stop it.

Jun 18, 2026 · 6 min read · Kash Gohil

Rezee is one workspace for the whole product development lifecycle - code, planning, CI/CD, chat, and docs. The payoff of code and planning living in one workspace is that they can talk to each other without a sync integration: write "Closes ACME-12" in a pull request, merge it, and ACME-12 moves to done on the board by itself. This post is about the small, deceptively tricky piece that makes that trustworthy - the parser that decides a PR closes an issue - tested against 500 real pull requests from major open-source projects, where the obvious implementation quietly closes the wrong thing.

Why is this harder than a regex?

The naive version is one line: find a closing keyword followed by an issue reference, anywhere in the text.

const re = /\b(close|closes|fix|fixes|resolve|resolves)\b\s*:?\s*(#\d+|[A-Z]{2,6}-\d+)/gi;

It works on Closes #12 and Fixes ACME-34, and it's wrong in exactly the ways that matter for an action that modifies your tracker without asking. Auto-close is a write triggered by text a human wrote for other humans, so the parser has to distinguish "I intend to close this" from "I mentioned closing this." Three real cases break the naive regex:

  • Code blocks. A PR that documents the auto-close feature itself - "add closes #123 to your PR body" - contains a closing keyword and a ref, in a code example, meaning nothing.
  • Negation. "This does not close #5, that's a separate fix" says the opposite of what the keyword says.
  • Prose that isn't intent. "Related to #22 (not closing)" mentions an issue without closing it.

Get this wrong and the failure is uniquely annoying: a merge silently marks the wrong issue done, someone notices days later, and trust in the whole automation erodes. So we measured how often the naive version is wrong, on real data.

The measured difference

First, a hand-labeled trap suite of 15 cases with known ground truth - the tricky phrasings above plus the legitimate ones (checklist items, colon forms, multiple refs):

Closing-ref parser quality on the trap suite Closing-ref parser quality on the trap suite robust (strip code + negation)naive (regex anywhere) 00.51 0.61110.81 precisionrecallf1 score

The naive parser has perfect recall and 0.64 precision: it never misses a real close, but a third of what it "closes" is a false positive - the code blocks and negations. The robust parser scores 1.0 on both. It makes exactly two corrections, and no more:

export function parseRobust(text: string): Ref[] {
	const clean = stripCode(text);              // 1. refs inside `code` are examples, not intent
	const re = /\b(close|closes|fix|fixes|resolve|resolves)\b\s*:?\s*(#\d+|[A-Z]{2,6}-\d+)/gi;
	const out: Ref[] = [];
	for (const m of clean.matchAll(re)) {
		const before = clean.slice(Math.max(0, m.index - 20), m.index);
		if (/\b(not|never|without|doesn't|won't|n't)\b/i.test(before)) continue;  // 2. reject negation
		out.push({ keyword: m[1].toLowerCase(), ref: m[2] });
	}
	return out;
}

Strip code fences and inline code before matching; reject a match with a negation word in the ~20 characters before it. That's it. Notably, we do not require the keyword to sit at the start of a clause - an earlier version did, and it dropped legitimate mid-sentence closes like "This resolves #99," which GitHub honors. The lesson of the trap suite was that clause position adds false negatives without removing a single false positive, because a bare keyword with no ref can't match anyway. The two corrections that matter are code and negation; everything else was over-engineering.

Does it matter on real pull requests?

A trap suite proves the parser handles cases you thought of. The honest test is real PRs full of cases you didn't. We ran both parsers over 500 closed pull requests from five major projects (cli/cli, Next.js, Deno, Prisma, Astro) - real bodies, real markdown, real mess:

  • Both parsers flagged closing refs in the same 142 PRs - so at the "does this PR close something" level, they agree on real data. Most PRs are unambiguous.
  • But counting individual references, the naive parser extracted 149 and the robust one 147. Those two extra refs are the whole point: closes #36139 and closes #22574, both found inside code blocks in real PRs, both of which the naive parser would have wrongly auto-closed on merge.

Two wrong closes in 500 PRs is a ~1.3% false-positive rate on extracted refs - low, but not zero, and it's a wrong write to someone's tracker every time it happens. The breakdown is the interesting part: on real data, all the naive parser's false positives were code blocks; the negated-closing-keyword case, which dominates the trap suite, essentially never occurs in the wild (people rarely write "does not close #5" with a real ref). We kept the negation check anyway because it's two lines and the day someone does write it, closing their issue would be a memorably bad bug - but the measurement told us which correction actually earns its keep.

What happens after the parse?

The parser is the hard part; the rest of auto-close is bookkeeping. On merge, makima parses the PR title and body (and the commit messages) for closing refs, records the link in an issue_prs table, and transitions each referenced issue to done - with the PR recorded in the issue's activity so the close is attributable, not mysterious. Because the issue's state is a status column and the transition is the same guarded update as everything else, the close is idempotent: re-processing the same merge (a retry, a replayed webhook) closes an already-closed issue to no effect. The parser decides which issues; the boring, safe machinery does the rest.

This is what "planning that closes itself" actually reduces to: not AI, not a heuristic, just a careful parser and a link table, made trustworthy by testing the parser against the messiest real input we could find rather than the clean examples that make any regex look correct.

FAQ

Why not just use GitHub's exact closing-keyword rules?

We nearly do - GitHub strips code blocks and matches the same keyword set, and we match that. The one place we're stricter is negation: GitHub famously does close an issue on "this does not close #5" (the keyword-plus-ref is all it checks), which is a long-standing gotcha. Since our corpus showed negated refs are rare, honoring the negation costs us nothing on real data and avoids the one surprising wrong-close. Where behavior is well-established (keyword set, code stripping) we match expectations; where it's a known wart, we quietly fix it.

Does it parse commit messages too, or only the PR body?

Both. A closing keyword in any commit on the PR, or in the PR title or body, counts - because people put "Fixes ACME-12" in whichever of those fits their habit. The parser runs over the concatenated text; the link table dedupes by issue, so "Closes #12" in both a commit and the body links #12 once, not twice.

What about a PR that says "Closes ACME-12" but shouldn't have?

Auto-close is reversible - reopen the issue and unlink the PR - because a parser, however careful, is inferring intent from prose and will occasionally infer wrong. The design accepts that the parse is a strong default, not a verdict: it moves the issue and records exactly why (the PR link in the activity feed), so a wrong close is one obvious click to undo, not a mystery to investigate. Trustworthy automation isn't automation that's never wrong; it's automation whose actions are legible and reversible.