Cumulative coverage and test gain

This page extends code coverage. It assumes you can already bind, run and ingest a window for Go or LLVM, and read documents with reflow coverage documents|gaps|lines|diff. To decide which gaps deserve a test and when to stop, see when to stop adding tests.

Before and after a new test

Coverage is process-wide: nothing attributes lines to a single flow. To learn what one new test adds without running the existing suite again, run only the new test in its own window, then compare two cumulative documents built from the same frozen binary: one from the baseline windows’ raw files, and one from those files plus the new window’s.

1. Record a baseline: run the existing tests in one or more run windows on one frozen instrumented binary. Keep every window’s cover directory unchanged after ingest, with its expected process count and apply keys. Build the baseline cumulative document from them and note its ID.
2. Find candidates: gaps --document <baseline> lists missed ranges. Read them with lines --with-source and pick reachable behaviour that no existing test asserts: an error path, an empty state, a boundary, a state transition.
3. Write the test. Record its purpose (the behaviour and risk) and its oracle (where the expected values come from) before you look at the delta.
4. Run only the new test once, in its own window: a fresh, empty cover directory for each producer, bound to the same frozen binaries (do not rebuild or re-sign them), with the same exclusions. Keep that directory too.
5. Build a new cumulative document from the baseline windows’ raw files plus the new window’s, then compare: diff --document <new cumulative> --base <baseline cumulative> --json. When the pair is likeForLike, the delta comes from the new test’s processes.

A diff between documents of different scenarios (for example one test’s window against another’s) is a scenario comparison. It can even be likeForLike, but it is never a test gain.

diff classifies the pair:

• likeForLike: same format, producer, toolchain, binary, mode, instrumented set, exclusions, tree and coverable code. gain reports the hit delta per metric for the whole comparison, plus newly hit and newly missed ranges per file on the current page (pageComplete is false while more pages follow). A net line delta of 0 can still hide lines that moved: check newlyHit and newlyMissed.
• descriptive: something other than tests changed (reasons such as tree_changed, toolchain_changed, binary_changed or exclude_changed). Totals are shown, gain is null, and the change must not be reported as test gain. To get a like-for-like pair, use the same instrumented binaries from the same commit on both sides; a rebuild can change the binary (binary_changed) even from the same source.
• Across commits (descriptive with only tree_changed, binary_changed or universe_changed), run diff in a checkout that holds both documents’ commits: the CLI maps unchanged lines through the commit diff and adds lineMap (status: "mapped") and, per file, mapped. mapped.newlyHit and mapped.newlyMissed are head lines whose text did not change but whose hit state did, so a flipped branch shows even when per-file totals are equal. mapped.changed, changedHit and changedMissed list the lines whose text changed and whether head ran them; they are not compared with base. This is behaviour change between two builds, never test gain. Without both commits, with other reasons, or with a dirty document, lineMap.status is unavailable with a reason, and changedLinesCommand names the changed-line form for the head document.
• Changed lines lead. A mapped result starts with changes: every file the commit diff changed, across all compare pages (up to 200 files; complete is false beyond), with changed, the head hit and missed lines, and the same file’s unchanged-line flips. It totals changedLines, instrumentable, hit and missed; notMeasured names changed files the head document did not instrument. Text output prints this block first. flips follows as secondary and covers this page only: unchanged-line flips in files the commit did not touch, as counts and the files with the most flips. Two documents from separate process windows ran different requests, retries and background work, so those flips are mostly run-to-run timing; do not read them as a regression without rerunning the same flows. --json keeps every range in mapped.
• A/A control. Rerun the base flows on the base build, ingest that run, and pass it with --control <document-id> (repeatable, up to 8; base=<id> is the same). Each control must be a clean document at the base document’s commit, tree and format (control_mismatch otherwise); a like-for-like pair refuses --control. The result then carries control right after changes: aa gives each control’s flips against base (the measured run-to-run noise), and beyond lists, in head line numbers and for every read file, the unchanged lines missed in base and in every control but hit in head (newlyHit), or hit in all of them but missed in head (newlyMissed). A line a control did not instrument is never beyond. These flips are behaviour signal: the change altered which code ran, even in files the commit did not touch. explained counts the other head flips. Text output prints Beyond the A/A control after the changed lines. Without the line mapping control.status is unavailable with the same reason as lineMap.
• Region flips. With a control, control.segments compares LLVM regions (kind: regions) or Go statement blocks (kind: blocks) the same way. This is the only signal when a changed decision sits inside one line, such as a one-line ternary: the line keeps running through its other branch, so no line flips, but the branch’s region stops. beyond.files[] lists each flip in head positions (line, column, endLine, endColumn; LLVM columns are 1-based bytes, end exclusive), lineFlipped when its line flipped too, and for LLVM the region’s source text from the local checkout (its first line, clipped, read through the same admission rules as coverage source). Regions touching changed lines are not compared. Go blocks keep no columns, so blocks sharing a line span and statement count are compared by how many ran (count of blocks). Text output prints Regions beyond the A/A control with newly missed <path>:<line>:<col>-<line>:<col> and the text for every flip on a line whose own coverage did not flip. A server that does not serve regions gives segments.status: unavailable (server_unsupported); lines still compare.
• Documents from different formats or producers are refused (format_mismatch, producer_kind_mismatch).

Cumulative documents from saved windows

There is no command that merges stored documents. Each ingest makes one document, and documents are never combined afterwards. Ingest does merge every raw file in its cover directory, so you can assemble one cumulative document per language from windows you already ran, with bind and ingest only and without running anything again. It is not a separate feature: record how each cumulative document was assembled.

Requirements

• One frozen binary. Every included window ran the exact same executable: the same file and SHA-256, never a rebuild or re-sign. Bind that file; ingest refuses binary_changed.
• The source it was built from. Run bind and ingest from a clean checkout of the commit the binary was built from. Go’s bind checks the binary’s VCS stamp against HEAD. LLVM binaries carry no stamp, so the source binding is whatever checkout you bind: choose the right commit.
• The same scope as the documents you compare. Use the same --collection, the same --exclude globs and, for LLVM, the same --path-strip.
• One fresh directory per language. Bind a new, empty directory outside the worktree, then copy the raw files in (bind refuses a directory that is not empty). Go and LLVM files never share a directory or a document.

Process count and batch keys

• --processes is the sum of each included window’s expected processes for that language, taken from what you launched (launch records, step states), never from the files you copied. Never lower it to match the files.
• Pass every included window’s apply keys with --batch: at most 64 keys, each with at most 1,000 runs (batch_too_large otherwise). Each key must have runs in the collection you pass (unknown_batch otherwise); a key whose apply ran nothing at all is left out and reported in emptyBatches. A flow that ran in several windows counts once in distinctFlows.
• If a window is missing a file it should have, leave that whole window out (its files, its share of the count and its keys) and say so, or stop. Never keep its keys while dropping its missing process.

Go

cd <clean checkout at the binary's commit>
reflow coverage bind --binary <frozen Go binary> --cover-dir <new-dir> \
  --collection <id> [--exclude <glob>]... --json
cp -p <window-1-dir>/covmeta.* <new-dir>/
cp -p <window-dir>/covcounters.* <new-dir>/   # every included window
reflow coverage ingest --cover-dir <new-dir> --processes <N> \
  --batch <key-1> --batch <key-2> --collection <id> --json

• Copy exactly one covmeta.* file. For one binary it is the same file in every window; compare them with cmp first, and stop if they differ.
• Copy the covcounters.* files with their names unchanged. Do not copy reflow-start.* markers or tmp.* files.
• Counter file names carry the process ID. If two windows reused the same process ID, ingest refuses duplicate_pid. Do not rename the files to get around it: split those windows into separate cumulative documents, or report the limitation.
• The document is exit-only: copied files cannot be verified at process start.

LLVM (Swift and other languages)

cd <clean checkout at the commit the binary was built from>
reflow coverage bind --format llvm-cov --binary <frozen Mach-O or ELF> \
  --cover-dir <new-dir> --path-strip <same prefix> --collection <id> --json
cp -p <window-dir>/<name>.profraw <new-dir>/<window>-<name>.profraw   # every profile
reflow coverage ingest --format llvm-cov --cover-dir <new-dir> --processes <N> \
  --batch <key-1> --batch <key-2> --collection <id> --json

• Process IDs repeat across windows, so prefix each copied file name with its window. Names may use letters, digits and ._%+- and must end in .profraw. Renaming changes only the name, never the contents.
• Ingest refuses any other file, a count other than N (profile_count_mismatch), and byte-identical profiles (profile_duplicate), even when they came from different processes. Copy each profile once; if two genuine profiles are identical, split the scope or report it.
• Reflow does not check that a profile came from the bound binary; llvm-cov drops what does not match. Copy only profiles written by launches of that exact executable.

Reading and labelling the result

• Read it like any document: gaps, files, lines, and diff --document <new cumulative> --base <previous cumulative>. With one frozen binary, one checkout and the same excludes, the pair is likeForLike; anything else is descriptive and never a test gain.
• Go and LLVM stay separate documents and separate diffs. Never blend them.
• Keep a record of each cumulative document: the source directories, the apply keys and the expected-process sums it was built from, and any window left out and why.