Make CI read your PR description

On September 17 I added a CI check to my company's repo. It doesn't run tests or look at code. It only reads the PR description, and turns red if the Validation, Risk, or Rollback section is missing.

The last line of that day's commit message:

Not in required checks yet (Actions stopped over billing)

In other words, on the day the check shipped, CI was down. It couldn't run on GitHub at all.

Why read the PR description

Agents open PRs eagerly and write long descriptions: which files changed, which functions were added, all neatly listed. But when I review a PR, what I most want to know is three other things. How did you confirm it works? What could it break? If it breaks, how do we undo it?

None of that is in the "Changes" section. If it's missing, I have to dig through the code and run things myself, and the long description saved me nothing.

You can write "every PR description needs Validation, Risk, and Rollback" in the rules file, of course. But a rule depends on the agent remembering it, and every lapse is a miss. And a section that just says "None" technically follows the rule. This is a job for a script: it doesn't forget, and it won't let something through because the description is long.

The script checks two things

First, all three sections exist. A heading containing "Validation," "Testing," or "Verification" counts as the validation section; same idea for Risk and Rollback. A single heading like "Risk and rollback" counts for both.

Second, a section can't be only a placeholder. If what's left after stripping punctuation is just "None," "N/A," "TODO," or "Same as above," it counts as empty. Template hints like <!-- how did you check --> don't count as content either; otherwise an agent could leave the template untouched and still pass.

What if there's genuinely nothing to write? My rules file says: don't delete the section, write why it doesn't apply. If you didn't test on a real device, say so in Validation.

For example, one PR for this site changed only copy, and its Risk section was one line:

Low. Copy changes only; removed one optional intro line.

It's short, but it's a judgment. The reader knows where the risk is and how big. "None" isn't a judgment.

Run it locally before you push

Back to that first day. When I looked into why CI had stopped, it wasn't unpaid bills. The month's free Actions quota had run out, and the Actions budget was set to $0, so every job was blocked once the quota was gone. The error said payment had failed, which looked like debt. Nothing was owed.

The quota reset in early October and CI came back. On October 12 it ran out again, and CI stayed down for a few days until I raised the budget from $0 to $20.

That's why the rule said from day one: run the check locally before opening a PR. While CI was down, that local run was the only gate. Even now the check isn't a required status check; a red result can still be merged. The local run is still the first gate.

Even with CI healthy, running it locally pays off. The agent runs it right before gh pr create, sees immediately what's missing, and fixes it on the spot. Pushing first and waiting for CI to go red is a round trip of several minutes.

Two more details from that commit are worth mentioning. A PR description is text anyone can write, so in GitHub Actions don't interpolate it into a shell command; pass it to the script through an environment variable, or someone can inject a command through the description. And for a check you plan to make required, don't use a workflow-level paths filter. A PR whose changes don't match the filter never runs the check, so it sits waiting forever and can never merge.

A version to copy

The script uses only the standard library. Put it at scripts/pr_body_lint.py:

#!/usr/bin/env python3
"""The PR description must have Validation / Risk / Rollback sections, with real content."""
import os
import re
import sys

SECTIONS = {
    "Validation": ("validation", "testing", "verification"),
    "Risk": ("risk",),
    "Rollback": ("rollback", "revert"),
}
PLACEHOLDERS = {"n/a", "na", "none", "todo", "tbd", "nothing", "same as above"}
HEADING = re.compile(r"^#{1,6}\s+(.+?)\s*$")


def split_sections(body):
    body = re.sub(r"<!--.*?-->", "", body, flags=re.S)  # template hints aren't content
    sections, in_fence = [], False
    for line in body.splitlines():
        if line.lstrip().startswith("```"):
            in_fence = not in_fence
        match = None if in_fence else HEADING.match(line)
        if match:
            sections.append((match.group(1).lower(), []))
        elif sections:
            sections[-1][1].append(line)
    return sections


def is_placeholder(lines):
    words = [re.sub(r"[\-*>`.:]+", "", line).strip().lower() for line in lines]
    return all(word in PLACEHOLDERS for word in words if word)


def main():
    if len(sys.argv) > 1:
        body = open(sys.argv[1], encoding="utf-8").read()
    else:
        body = os.environ.get("PR_BODY", "")
    sections = split_sections(body)
    errors = []
    for name, aliases in SECTIONS.items():
        found = [lines for title, lines in sections if any(a in title for a in aliases)]
        if not found:
            errors.append(f"Missing a {name} section")
        elif all(is_placeholder(lines) for lines in found):
            errors.append(f"{name} has only placeholders; if it doesn't apply, say why")
    for error in errors:
        print(error)
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main())

Wire it into CI like this. The description goes through an environment variable, and editing the description re-runs the check:

name: pr-body
on:
  pull_request:
    types: [opened, edited, synchronize, reopened]
jobs:
  pr-body:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 scripts/pr_body_lint.py
        env:
          PR_BODY: ${{ github.event.pull_request.body }}

Then add this to the rules file so the agent runs it locally first:

- Every PR description has Validation, Risk, and Rollback sections; if
  one doesn't apply, say why; if a check wasn't done, say so
- Before opening a PR, run `python3 scripts/pr_body_lint.py body.md`
  and only run `gh pr create` once it passes

The script only checks shape. If Validation says "all tests pass," it can't tell whether the tests ran or the agent just said so. That takes other rules, like requiring pasted output for failed tests in the wrap-up report.

The check that couldn't run on its first day now runs on PRs on GitHub, though a red result still can't block a merge. The only thing that can stop a bad description before it merges is still the local run.