Swift tests and coverage (prototype)

SwiftVM is a private source-installed prototype, absent from the public CLI bundle. Its host tool svm builds its own fixture Swift package and runs Swift Testing suites inside a disposable macOS guest, then transfers results, raw coverage profiles and build products back with their hashes checked. This fixture coverage remains a diagnostic: requesting it does not produce a passing gate. See source installation.

The separate vm.guest_run Reflow command runs staged jobs, including native app journeys and user-supplied build/test scripts, in fresh guest clones. It has no dedicated Swift Testing adapter or built-in validation of arbitrary packages’ test results. The fixture validation described below belongs to svm run.

One provider, SwiftVM (flow alias vm), covers this page and macOS VM, which describes templates, guest jobs, display and host setup. Instrumented app coverage can be exported from guest jobs and ingested on the host with Reflow coverage tools.

Host requirements

The macOS VM host requirements apply in full. In short:

• An Apple Silicon Mac on physical hardware (no nested macOS virtualization), running macOS 27, with guests on macOS 27 or later.
• A logged-in graphical session on the host. Virtual machines run on an offscreen display by default, so nothing appears on the desktop, but the session’s window server is still needed.
• Command Line Tools 27.0 on the host (Swift 6.4, SDK 27.0); provisioning installs its own copy in the guest. About 100 GB of free disk for first-time setup.
• The host tool signed with the com.apple.security.virtualization entitlement.
• Direct HTTPS from the guest to Apple’s software update servers while the template is provisioned.

How tests run

The guest has Command Line Tools, not Xcode, so tests use Swift Testing only. XCTest and .xcresult bundles are not supported.

Inside the guest, the prototype builds the package with Swift Build and coverage enabled. In essence (the real call also passes the package and scratch paths, the Swift Testing macro plugin path and rpaths to the Command Line Tools testing libraries):

swift build --build-system swiftbuild --build-tests --enable-code-coverage

The tests then run through the toolchain’s swiftpm-testing-helper with --testing-library swift-testing, writing xunit XML.

So far the prototype runs its own fixture package: a small AppKit app with a passing suite, a suite that fails on purpose and a slow suite for cancellation and timeout checks. Result validation expects that fixture’s exact test identities, so this fixture-specific result validator does not accept your own package. Generic vm.guest_run jobs can run your build and test scripts separately.

Results

Test results are accepted only in a pinned xunit dialect from the Command Line Tools 27.0 / Swift 6.4 helper. Test identities must match exactly and summaries must be consistent; errors, skipped tests, duplicates and malformed counts are refused. If a template records a different Command Line Tools or Swift version, the run warns and adds a limitation to its report.

Artifacts are listed by the guest with their sizes and SHA-256 hashes and re-hashed on the host. A failed or mismatched transfer is recorded as corrupt and its partial file is deleted. The selected tests must come with the test bundle they were linked into. This proves the artifacts were transferred intact and that the evidence is consistent. It does not prove the guest is honest: the guest can fabricate results, and untrusted job code is not supported.

The run’s verdict and exit code are described on the macOS VM page.

Coverage

When coverage is requested:

1. Each test process writes a raw profile named <name>-%p.profraw.
2. In the guest, llvm-profdata merge -sparse merges them and llvm-cov export -summary-only writes a summary.
3. On the host, llvm-cov export -summary-only runs again over the guest’s merged profile and binary, and its summary is validated.

Command Line Tools 27’s llvm-cov writes export format 3.0.1, which is the version the prototype accepts.

A run that requests coverage never passes: at best it ends incomplete (exit 4), even when every test and coverage-integrity check passes. A failed check, unconfirmed cleanup or an interrupt gives that run’s usual exit code instead. There is no coverage policy, no default threshold, and no complete record tying the source, the build and the profiles together. Missing policy or provenance is never treated as a pass.

Coverage has one more trust limit: the host’s llvm-cov parses profiles and binaries produced inside the guest, and that parsing is not sandboxed.

Reflow code coverage

The fixture’s summary-only coverage export is not a canonical Reflow coverage document: it lacks full per-file and per-function data, and no automatic fixture adapter exists. Its incomplete coverage verdict remains unchanged.

For vm.guest_run, build an instrumented app, retain its exact binaries and profiles under workspace out/, then bind and ingest the exported coverage on the host. This manual path has ingested coverage from native Gigahog guest journeys. A guest job passing does not itself ingest coverage or prove a coverage policy. See Swift and other LLVM languages for the required full llvm-cov export and process counts. The CLI accepts export formats 2.0.1 and 3.0.1, including Command Line Tools 27’s format.

Recorded evidence

On one Apple Silicon host with a freshly provisioned macOS 27.0.1 template (Command Line Tools 27.0, Swift 6.4), on 2026-10-15, built from an earlier version of the prototype source:

• Tests and coverage (passing, failing, cancellation and timeout suites): every other required check passed (the deliberately failing suite failed as expected; cancellation and timeout were observed), and profiles, exports and coverage integrity validated. The run ended incomplete (exit 4) in 200 seconds because coverage policy and exact-build provenance were unproved. The passing suite’s xunit shows 3 tests and 0 failures.
• The default gate (passing suite and GUI proof): passed, exit 0, in 202 seconds.
• Everything selected: incomplete, exit 4, in 488 seconds.

The host was heavily loaded by other work during these runs, so the times are not performance figures.

A later build, on an APFS copy of that template, passed the default gate again with exit 0 in 187 seconds, including the GUI proof described on macOS VM. These fixture measurements predate the source-installed Reflow guest-journey workflow and do not qualify every later source revision. Native guest journeys and host coverage ingestion were subsequently exercised through vm.guest_run; see recorded evidence.

These are prototype observations from a few runs, not a release qualification.

Not yet supported

• A public SwiftVM release or installation without private source access.
• A dedicated Reflow command for Swift Testing suites or validated arbitrary package results; generic guest jobs are available separately.
• XCTest, .xcresult bundles and XCTest coverage, end to end.
• A passing fixture coverage gate, fixture coverage policies or thresholds, or automatic ingestion of its summary-only export.
• A complete exact-build fixture coverage gate tying source, build, test processes and profiles together; generic guest-job export hashes do not establish that gate.
• An offline or proxied Command Line Tools installation during provisioning.