Validate FreeBSD base rollback workflow

2026-04-03 10:42:38 +02:00
parent 72f89c51b5
commit 03fbd9bf08
6 changed files with 784 additions and 0 deletions
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -3538,3 +3538,89 @@ Current assessment:
 - Phase 15.1 is complete
 - Fruix now models the FreeBSD base as an explicit declarative system input instead of leaving it implicit in the builder host alone
 - the next step is to use that new declaration to prove side-by-side base versions and rollback-friendly rebuild/redeploy behavior
+
+## 2026-04-03 — Phase 15.2: validated side-by-side base versions and rollback-friendly redeploy
+
+Completed work:
+
+- wrote the Phase 15.2 report:
+  - `docs/reports/phase15-base-upgrades-freebsd.md`
+- added validation harnesses:
+  - `tests/system/run-phase15-base-coexistence.sh`
+  - `tests/system/run-phase15-base-rollback-qemu.sh`
+  - `tests/system/run-phase15-base-rollback-xcpng.sh`
+- used two explicit declarative base identities against the current validated native Phase 14 package split:
+  - current base:
+    - `name=stable-default`
+    - `version-label=15.0-STABLE`
+    - `release=15.0-STABLE`
+    - `branch=stable/15`
+  - candidate base:
+    - `name=stable-canary`
+    - `version-label=15.0-STABLE-p1`
+    - `release=15.0-STABLE`
+    - `branch=stable/15`
+- both declarations still use the same local `/usr/src`, but now produce distinct declared base/store/deployment identities
+
+Validation:
+
+- side-by-side base-coexistence harness passes:
+  - `tests/system/run-phase15-base-coexistence.sh`
+  - workdir: `/tmp/phase15-2-coexist-1775202833`
+  - result: `PASS phase15-base-coexistence`
+  - confirmed:
+    - `current_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd`
+    - `candidate_closure=/frx/store/dc40b1b7a76084e140d0457f3b7f6c5d4acc185f0d6cee0b161c9775d5fb3bec-fruix-system-fruix-freebsd`
+    - `current_base_version_label=15.0-STABLE`
+    - `candidate_base_version_label=15.0-STABLE-p1`
+    - `side_by_side_base_versions=ok`
+    - `rollback_rebuild_path=ok`
+  - this also confirmed that:
+    - both closures exist side by side in `/frx/store`
+    - rebuilding the current declaration returns the exact original current closure path
+    - current native base stores remain separate from candidate native base stores
+- local QEMU rollback harness passes:
+  - `tests/system/run-phase15-base-rollback-qemu.sh`
+  - workdir: `/tmp/phase15-2-qemu2-1775204321`
+  - result: `PASS phase15-base-rollback-qemu`
+  - validation sequence:
+    1. boot current base
+    2. boot candidate base
+    3. boot current base again
+  - confirmed:
+    - `current_first_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd`
+    - `candidate_closure=/frx/store/dc40b1b7a76084e140d0457f3b7f6c5d4acc185f0d6cee0b161c9775d5fb3bec-fruix-system-fruix-freebsd`
+    - `rollback_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd`
+    - `current_base_version_label=15.0-STABLE`
+    - `candidate_base_version_label=15.0-STABLE-p1`
+    - `rollback_base_version_label=15.0-STABLE`
+    - `base_rollforward_and_rollback=ok`
+- real XCP-ng rollback harness passes:
+  - `tests/system/run-phase15-base-rollback-xcpng.sh`
+  - workdir: `/tmp/phase15-2-xcpng-1775204839`
+  - result: `PASS phase15-base-rollback-xcpng`
+  - validation sequence:
+    1. boot candidate base on the approved VM/VDI
+    2. boot current base again on the same approved VM/VDI
+  - confirmed:
+    - `candidate_closure=/frx/store/dc40b1b7a76084e140d0457f3b7f6c5d4acc185f0d6cee0b161c9775d5fb3bec-fruix-system-fruix-freebsd`
+    - `rollback_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd`
+    - `candidate_base_version_label=15.0-STABLE-p1`
+    - `rollback_base_version_label=15.0-STABLE`
+    - `vm_id=90490f2e-e8fc-4b7a-388e-5c26f0157289`
+    - `vdi_id=0f1f90d3-48ca-4fa2-91d8-fc6339b95743`
+    - `base_rollforward_and_rollback=ok`
+  - guest invariants stayed intact for both boots:
+    - `shepherd_pid=1`
+    - `sshd_status=running`
+    - `compat_prefix_shims=absent`
+    - `guile_module_smoke=ok`
+
+Current assessment:
+
+- Phase 15.2 is complete
+- Fruix now supports a real upgrade-style FreeBSD base workflow at the store/deployment level:
+  - explicit current vs. candidate base declarations
+  - side-by-side native base outputs in `/frx/store`
+  - rollback to the earlier closure without mutating it in place
+- the remaining Phase 15 work is to document the evidence-based decision on whether self-hosted base builds should be the next step, or whether host-built native base artifacts should remain the near-term path while reproducibility/source acquisition improve
--- a/docs/reports/phase15-base-upgrades-freebsd.md
+++ b/docs/reports/phase15-base-upgrades-freebsd.md
@@ -0,0 +1,179 @@
+# Phase 15.2: side-by-side native base versions and rollback-friendly redeploy
+
+Date: 2026-04-03
+
+## Goal
+
+Phase 15.2 demonstrated that Fruix can keep at least two distinct declarative FreeBSD base builds in `/frx/store` at the same time and switch between them through the normal system rebuild/image/boot flow.
+
+For this first upgrade-story validation, both declared bases still point at the same local `/usr/src`, but they carry distinct declarative version labels:
+
+- current base: `15.0-STABLE`
+- candidate base: `15.0-STABLE-p1`
+
+That is enough to prove the Fruix properties needed here:
+
+- distinct content-addressed outputs
+- side-by-side coexistence
+- no in-place mutation of the older base closure
+- rollback to the earlier closure using the normal deployment path
+
+## New files
+
+Added:
+
+- `tests/system/run-phase15-base-coexistence.sh`
+- `tests/system/run-phase15-base-rollback-qemu.sh`
+- `tests/system/run-phase15-base-rollback-xcpng.sh`
+
+## Validation model
+
+### Current base declaration
+
+```scheme
+(freebsd-base
+  #:name "stable-default"
+  #:version-label "15.0-STABLE"
+  #:release "15.0-STABLE"
+  #:branch "stable/15"
+  ...)
+```
+
+### Candidate base declaration
+
+```scheme
+(freebsd-base
+  #:name "stable-canary"
+  #:version-label "15.0-STABLE-p1"
+  #:release "15.0-STABLE"
+  #:branch "stable/15"
+  ...)
+```
+
+Both declarations use the same validated native Phase 14 package composition:
+
+- kernel from `freebsd-native-kernel-for`
+- bootloader from `freebsd-native-bootloader-for`
+- runtime from `freebsd-native-system-packages-for`
+- `shepherd-pid1`
+
+## Side-by-side build validation
+
+Passing run:
+
+- `PASS phase15-base-coexistence`
+- workdir: `/tmp/phase15-2-coexist-1775202833`
+
+Confirmed:
+
+```text
+current_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd
+candidate_closure=/frx/store/dc40b1b7a76084e140d0457f3b7f6c5d4acc185f0d6cee0b161c9775d5fb3bec-fruix-system-fruix-freebsd
+current_base_version_label=15.0-STABLE
+candidate_base_version_label=15.0-STABLE-p1
+side_by_side_base_versions=ok
+rollback_rebuild_path=ok
+```
+
+Current native base stores:
+
+```text
+/frx/store/d9785661ea4829d51fbf545c2607a5691af2cc33c8ef3cd44de7ad5626685098-freebsd-native-kernel-15.0-STABLE
+/frx/store/b448c822302ccdfb2f06da811fb224a044c51a9935bbfcd77a71a25d02f228f1-freebsd-native-bootloader-15.0-STABLE
+/frx/store/ac3ba684020e70d3c76e593fd687cef8ab5e148958baabb477b7ef3d2647c5cd-freebsd-native-runtime-15.0-STABLE
+```
+
+Candidate native base stores:
+
+```text
+/frx/store/05bee8ffbe8c43242ffd97da4dc305f2921612a660cbcb48c3a3536bfac07079-freebsd-native-kernel-15.0-STABLE-p1
+/frx/store/8955f1bfe89321e6e1e628c59376f2092547523f48a773974cc259963adac184-freebsd-native-bootloader-15.0-STABLE-p1
+/frx/store/30314f17fd8ff4a1a3eff31c8c5048f15f67c46d1132d5b8c45fd9768742665e-freebsd-native-runtime-15.0-STABLE-p1
+```
+
+Important result:
+
+- the older current closure stayed in `/frx/store`
+- the candidate closure appeared beside it
+- rebuilding the current declaration returned the exact original current closure path again
+
+## Local QEMU rollback validation
+
+Passing run:
+
+- `PASS phase15-base-rollback-qemu`
+- workdir: `/tmp/phase15-2-qemu2-1775204321`
+
+Validation sequence:
+
+1. boot current base
+2. boot candidate base
+3. boot current base again
+
+Confirmed:
+
+```text
+current_first_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd
+candidate_closure=/frx/store/dc40b1b7a76084e140d0457f3b7f6c5d4acc185f0d6cee0b161c9775d5fb3bec-fruix-system-fruix-freebsd
+rollback_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd
+current_base_version_label=15.0-STABLE
+candidate_base_version_label=15.0-STABLE-p1
+rollback_base_version_label=15.0-STABLE
+base_rollforward_and_rollback=ok
+```
+
+This showed that the booted system could move forward to the candidate base and then return to the earlier closure without mutating it in place.
+
+## Real XCP-ng rollback validation
+
+Passing run:
+
+- `PASS phase15-base-rollback-xcpng`
+- workdir: `/tmp/phase15-2-xcpng-1775204839`
+
+Validation sequence:
+
+1. boot candidate base on the approved VM/VDI
+2. boot current base again on the same approved VM/VDI
+
+Confirmed:
+
+```text
+candidate_closure=/frx/store/dc40b1b7a76084e140d0457f3b7f6c5d4acc185f0d6cee0b161c9775d5fb3bec-fruix-system-fruix-freebsd
+rollback_closure=/frx/store/9f57ecc6481e271811ceb53ac21a3b2aef4ef329f82b7d4788622315db1f0e43-fruix-system-fruix-freebsd
+candidate_base_version_label=15.0-STABLE-p1
+rollback_base_version_label=15.0-STABLE
+vm_id=90490f2e-e8fc-4b7a-388e-5c26f0157289
+vdi_id=0f1f90d3-48ca-4fa2-91d8-fc6339b95743
+base_rollforward_and_rollback=ok
+```
+
+Both boots also preserved the already-hardened guest properties:
+
+- `shepherd_pid=1`
+- `sshd_status=running`
+- `compat_prefix_shims=absent`
+- `guile_module_smoke=ok`
+
+## Result
+
+Phase 15.2 is complete.
+
+Fruix now has a real declarative rebuild/redeploy/rollback story for the FreeBSD base at the store/model layer:
+
+- two declared base versions can coexist side by side in `/frx/store`
+- the candidate deployment does not overwrite the current one in place
+- rebuilding the earlier declaration returns to the earlier closure path
+- the same story works both locally under QEMU and on the approved XCP-ng VM/VDI path
+
+## Scope note
+
+This first upgrade-story validation still uses the same local `/usr/src` as the underlying source tree for both declarations.  What changed is the declared base identity and therefore the store/model/deployment identity.
+
+That is sufficient for this phase because the requirement was to establish the upgrade semantics:
+
+- explicit base declaration
+- side-by-side outputs
+- rollback-friendly closures
+
+The next improvement beyond Phase 15 would be to make acquiring or selecting distinct source trees/releases more reproducible and less tied to a single host checkout.
--- a/tests/system/run-phase15-base-coexistence.sh
+++ b/tests/system/run-phase15-base-coexistence.sh
@@ -0,0 +1,192 @@
+#!/bin/sh
+set -eu
+
+project_root=${PROJECT_ROOT:-$(pwd)}
+script_dir=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
+fruix_cmd=$project_root/bin/fruix
+os_template=${OS_TEMPLATE:-$script_dir/phase15-declarative-base-pid1-operating-system.scm.in}
+system_name=${SYSTEM_NAME:-phase15-operating-system}
+store_dir=${STORE_DIR:-/frx/store}
+current_base_name=${CURRENT_BASE_NAME:-stable-default}
+current_base_version=${CURRENT_BASE_VERSION:-15.0-STABLE}
+current_base_release=${CURRENT_BASE_RELEASE:-15.0-STABLE}
+current_base_branch=${CURRENT_BASE_BRANCH:-stable/15}
+candidate_base_name=${CANDIDATE_BASE_NAME:-stable-canary}
+candidate_base_version=${CANDIDATE_BASE_VERSION:-15.0-STABLE-p1}
+candidate_base_release=${CANDIDATE_BASE_RELEASE:-15.0-STABLE}
+candidate_base_branch=${CANDIDATE_BASE_BRANCH:-stable/15}
+metadata_target=${METADATA_OUT:-}
+root_authorized_key_file=${ROOT_AUTHORIZED_KEY_FILE:-$HOME/.ssh/id_ed25519.pub}
+
+[ -x "$fruix_cmd" ] || {
+  echo "fruix command is not executable: $fruix_cmd" >&2
+  exit 1
+}
+[ -f "$os_template" ] || {
+  echo "missing operating-system template: $os_template" >&2
+  exit 1
+}
+[ -f "$root_authorized_key_file" ] || {
+  echo "missing root authorized key file: $root_authorized_key_file" >&2
+  exit 1
+}
+
+cleanup=0
+if [ -n "${WORKDIR:-}" ]; then
+  workdir=$WORKDIR
+  mkdir -p "$workdir"
+else
+  workdir=$(mktemp -d /tmp/fruix-phase15-base-coexistence.XXXXXX)
+  cleanup=1
+fi
+if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then
+  cleanup=0
+fi
+
+cleanup_workdir() {
+  if [ "$cleanup" -eq 1 ]; then
+    rm -rf "$workdir" 2>/dev/null || sudo rm -rf "$workdir"
+  fi
+}
+trap cleanup_workdir EXIT INT TERM
+
+render_os() {
+  output=$1
+  base_name=$2
+  base_version=$3
+  base_release=$4
+  base_branch=$5
+  root_authorized_key=$(tr -d '\n' < "$root_authorized_key_file")
+  sed \
+    -e "s|__BASE_NAME__|$base_name|g" \
+    -e "s|__BASE_VERSION_LABEL__|$base_version|g" \
+    -e "s|__BASE_RELEASE__|$base_release|g" \
+    -e "s|__BASE_BRANCH__|$base_branch|g" \
+    -e "s|__ROOT_AUTHORIZED_KEY__|$root_authorized_key|g" \
+    "$os_template" > "$output"
+}
+
+action_env() {
+  sudo env \
+    HOME="$HOME" \
+    GUILE_AUTO_COMPILE=0 \
+    FRUIX_FREEBSD_BUILD_JOBS="${FRUIX_FREEBSD_BUILD_JOBS:-8}" \
+    GUIX_SOURCE_DIR="${GUIX_SOURCE_DIR:-$HOME/repos/guix}" \
+    GUILE_BIN="${GUILE_BIN:-/tmp/guile-freebsd-validate-install/bin/guile}" \
+    GUILE_EXTRA_PREFIX="${GUILE_EXTRA_PREFIX:-/tmp/guile-gnutls-freebsd-validate-install}" \
+    SHEPHERD_PREFIX="${SHEPHERD_PREFIX:-/tmp/shepherd-freebsd-validate-install}" \
+    "$@"
+}
+
+build_os() {
+  os_file=$1
+  out_file=$2
+  action_env "$fruix_cmd" system build "$os_file" --system "$system_name" --store "$store_dir" >"$out_file"
+}
+
+current_os=$workdir/current-operating-system.scm
+candidate_os=$workdir/candidate-operating-system.scm
+current_out_a=$workdir/current-build-a.txt
+current_out_b=$workdir/current-build-b.txt
+candidate_out=$workdir/candidate-build.txt
+metadata_file=$workdir/phase15-base-coexistence-metadata.txt
+
+render_os "$current_os" "$current_base_name" "$current_base_version" "$current_base_release" "$current_base_branch"
+render_os "$candidate_os" "$candidate_base_name" "$candidate_base_version" "$candidate_base_release" "$candidate_base_branch"
+
+build_os "$current_os" "$current_out_a"
+build_os "$candidate_os" "$candidate_out"
+build_os "$current_os" "$current_out_b"
+
+current_closure=$(sed -n 's/^closure_path=//p' "$current_out_a")
+current_closure_rebuild=$(sed -n 's/^closure_path=//p' "$current_out_b")
+candidate_closure=$(sed -n 's/^closure_path=//p' "$candidate_out")
+current_native_stores=$(sed -n 's/^native_base_stores=//p' "$current_out_a")
+candidate_native_stores=$(sed -n 's/^native_base_stores=//p' "$candidate_out")
+current_host_count=$(sed -n 's/^host_base_store_count=//p' "$current_out_a")
+candidate_host_count=$(sed -n 's/^host_base_store_count=//p' "$candidate_out")
+current_base_version_out=$(sed -n 's/^freebsd_base_version_label=//p' "$current_out_a")
+candidate_base_version_out=$(sed -n 's/^freebsd_base_version_label=//p' "$candidate_out")
+current_base_file=$(sed -n 's/^freebsd_base_file=//p' "$current_out_a")
+candidate_base_file=$(sed -n 's/^freebsd_base_file=//p' "$candidate_out")
+
+[ -n "$current_closure" ] || { echo "missing current closure" >&2; exit 1; }
+[ -n "$candidate_closure" ] || { echo "missing candidate closure" >&2; exit 1; }
+[ "$current_closure" = "$current_closure_rebuild" ] || {
+  echo "current closure path was not reproducible: $current_closure != $current_closure_rebuild" >&2
+  exit 1
+}
+[ "$current_closure" != "$candidate_closure" ] || {
+  echo "current and candidate closures unexpectedly match" >&2
+  exit 1
+}
+[ "$current_host_count" = 0 ] || { echo "current build has host base stores" >&2; exit 1; }
+[ "$candidate_host_count" = 0 ] || { echo "candidate build has host base stores" >&2; exit 1; }
+[ "$current_base_version_out" = "$current_base_version" ] || { echo "unexpected current base version label" >&2; exit 1; }
+[ "$candidate_base_version_out" = "$candidate_base_version" ] || { echo "unexpected candidate base version label" >&2; exit 1; }
+[ -f "$current_base_file" ] || { echo "missing current base file" >&2; exit 1; }
+[ -f "$candidate_base_file" ] || { echo "missing candidate base file" >&2; exit 1; }
+
+for path in "$current_closure" "$candidate_closure"; do
+  [ -d "$path" ] || {
+    echo "expected closure directory missing: $path" >&2
+    exit 1
+  }
+done
+
+printf '%s\n' "$current_native_stores" | tr ',' '\n' | grep "freebsd-native-kernel-$current_base_version$" >/dev/null || {
+  echo "current native stores do not contain the expected kernel version label" >&2
+  exit 1
+}
+printf '%s\n' "$candidate_native_stores" | tr ',' '\n' | grep "freebsd-native-kernel-$candidate_base_version$" >/dev/null || {
+  echo "candidate native stores do not contain the expected kernel version label" >&2
+  exit 1
+}
+[ "$current_native_stores" != "$candidate_native_stores" ] || {
+  echo "current and candidate native store sets unexpectedly match" >&2
+  exit 1
+}
+
+grep -F "(version-label . \"$current_base_version\")" "$current_base_file" >/dev/null || {
+  echo "current base file missing version label" >&2
+  exit 1
+}
+grep -F "(version-label . \"$candidate_base_version\")" "$candidate_base_file" >/dev/null || {
+  echo "candidate base file missing version label" >&2
+  exit 1
+}
+
+current_closure_base=$(basename "$current_closure")
+candidate_closure_base=$(basename "$candidate_closure")
+cat >"$metadata_file" <<EOF
+workdir=$workdir
+current_os=$current_os
+candidate_os=$candidate_os
+current_closure=$current_closure
+current_closure_rebuild=$current_closure_rebuild
+current_closure_base=$current_closure_base
+candidate_closure=$candidate_closure
+candidate_closure_base=$candidate_closure_base
+current_native_stores=$current_native_stores
+candidate_native_stores=$candidate_native_stores
+current_freebsd_base_file=$current_base_file
+candidate_freebsd_base_file=$candidate_base_file
+current_base_version_label=$current_base_version_out
+candidate_base_version_label=$candidate_base_version_out
+side_by_side_base_versions=ok
+rollback_rebuild_path=ok
+EOF
+
+if [ -n "$metadata_target" ]; then
+  mkdir -p "$(dirname "$metadata_target")"
+  cp "$metadata_file" "$metadata_target"
+fi
+
+printf 'PASS phase15-base-coexistence\n'
+printf 'Work directory: %s\n' "$workdir"
+printf 'Metadata file: %s\n' "$metadata_file"
+if [ -n "$metadata_target" ]; then
+  printf 'Copied metadata to: %s\n' "$metadata_target"
+fi
+printf '%s\n' '--- metadata ---'
+cat "$metadata_file"
--- a/tests/system/run-phase15-base-rollback-qemu.sh
+++ b/tests/system/run-phase15-base-rollback-qemu.sh
@@ -0,0 +1,157 @@
+#!/bin/sh
+set -eu
+
+repo_root=${PROJECT_ROOT:-$(pwd)}
+script_dir=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
+os_template=${OS_TEMPLATE:-$script_dir/phase15-declarative-base-pid1-operating-system.scm.in}
+system_name=${SYSTEM_NAME:-phase15-operating-system}
+disk_capacity=${DISK_CAPACITY:-8g}
+root_size=${ROOT_SIZE:-6g}
+current_base_name=${CURRENT_BASE_NAME:-stable-default}
+current_base_version=${CURRENT_BASE_VERSION:-15.0-STABLE}
+current_base_release=${CURRENT_BASE_RELEASE:-15.0-STABLE}
+current_base_branch=${CURRENT_BASE_BRANCH:-stable/15}
+candidate_base_name=${CANDIDATE_BASE_NAME:-stable-canary}
+candidate_base_version=${CANDIDATE_BASE_VERSION:-15.0-STABLE-p1}
+candidate_base_release=${CANDIDATE_BASE_RELEASE:-15.0-STABLE}
+candidate_base_branch=${CANDIDATE_BASE_BRANCH:-stable/15}
+metadata_target=${METADATA_OUT:-}
+cleanup=0
+
+if [ -n "${WORKDIR:-}" ]; then
+  workdir=$WORKDIR
+  mkdir -p "$workdir"
+else
+  workdir=$(mktemp -d /tmp/fruix-phase15-base-rollback-qemu.XXXXXX)
+  cleanup=1
+fi
+if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then
+  cleanup=0
+fi
+
+metadata_file=$workdir/phase15-base-rollback-qemu-metadata.txt
+current_template=$workdir/current-template.scm.in
+candidate_template=$workdir/candidate-template.scm.in
+
+cleanup_workdir() {
+  if [ "$cleanup" -eq 1 ]; then
+    rm -rf "$workdir" 2>/dev/null || sudo rm -rf "$workdir"
+  fi
+}
+trap cleanup_workdir EXIT INT TERM
+
+render_template() {
+  output=$1
+  base_name=$2
+  base_version=$3
+  base_release=$4
+  base_branch=$5
+  sed \
+    -e "s|__BASE_NAME__|$base_name|g" \
+    -e "s|__BASE_VERSION_LABEL__|$base_version|g" \
+    -e "s|__BASE_RELEASE__|$base_release|g" \
+    -e "s|__BASE_BRANCH__|$base_branch|g" \
+    "$os_template" > "$output"
+}
+
+render_template "$current_template" "$current_base_name" "$current_base_version" "$current_base_release" "$current_base_branch"
+render_template "$candidate_template" "$candidate_base_name" "$candidate_base_version" "$candidate_base_release" "$candidate_base_branch"
+
+run_boot() {
+  name=$1
+  template=$2
+  metadata_out=$3
+  KEEP_WORKDIR=1 WORKDIR="$workdir/$name" METADATA_OUT="$metadata_out" \
+    OS_TEMPLATE="$template" SYSTEM_NAME="$system_name" DISK_CAPACITY="$disk_capacity" ROOT_SIZE="$root_size" \
+    "$repo_root/tests/system/run-phase11-shepherd-pid1-qemu.sh" >/dev/null
+}
+
+current_first_metadata=$workdir/current-first-metadata.txt
+candidate_metadata=$workdir/candidate-metadata.txt
+rollback_metadata=$workdir/rollback-metadata.txt
+
+run_boot current-first "$current_template" "$current_first_metadata"
+run_boot candidate "$candidate_template" "$candidate_metadata"
+run_boot rollback "$current_template" "$rollback_metadata"
+
+current_first_phase8=$(sed -n 's/^phase8_metadata=//p' "$current_first_metadata")
+candidate_phase8=$(sed -n 's/^phase8_metadata=//p' "$candidate_metadata")
+rollback_phase8=$(sed -n 's/^phase8_metadata=//p' "$rollback_metadata")
+
+current_first_closure=$(sed -n 's/^closure_path=//p' "$current_first_metadata")
+candidate_closure=$(sed -n 's/^closure_path=//p' "$candidate_metadata")
+rollback_closure=$(sed -n 's/^closure_path=//p' "$rollback_metadata")
+
+current_first_version=$(sed -n 's/^freebsd_base_version_label=//p' "$current_first_phase8")
+candidate_version=$(sed -n 's/^freebsd_base_version_label=//p' "$candidate_phase8")
+rollback_version=$(sed -n 's/^freebsd_base_version_label=//p' "$rollback_phase8")
+
+current_first_shepherd_pid=$(sed -n 's/^shepherd_pid=//p' "$current_first_metadata")
+candidate_shepherd_pid=$(sed -n 's/^shepherd_pid=//p' "$candidate_metadata")
+rollback_shepherd_pid=$(sed -n 's/^shepherd_pid=//p' "$rollback_metadata")
+current_first_sshd=$(sed -n 's/^sshd_status=//p' "$current_first_metadata")
+candidate_sshd=$(sed -n 's/^sshd_status=//p' "$candidate_metadata")
+rollback_sshd=$(sed -n 's/^sshd_status=//p' "$rollback_metadata")
+
+[ "$current_first_version" = "$current_base_version" ] || { echo "unexpected current version label" >&2; exit 1; }
+[ "$candidate_version" = "$candidate_base_version" ] || { echo "unexpected candidate version label" >&2; exit 1; }
+[ "$rollback_version" = "$current_base_version" ] || { echo "unexpected rollback version label" >&2; exit 1; }
+[ "$current_first_closure" != "$candidate_closure" ] || { echo "candidate closure matches current closure" >&2; exit 1; }
+[ "$current_first_closure" = "$rollback_closure" ] || { echo "rollback closure did not return to the original current closure" >&2; exit 1; }
+for value in "$current_first_shepherd_pid" "$candidate_shepherd_pid" "$rollback_shepherd_pid"; do
+  [ "$value" = 1 ] || { echo "shepherd was not PID 1" >&2; exit 1; }
+done
+for value in "$current_first_sshd" "$candidate_sshd" "$rollback_sshd"; do
+  [ "$value" = running ] || { echo "sshd is not running in one of the rollback boots" >&2; exit 1; }
+done
+
+current_first_native=$(sed -n 's/^native_base_stores=//p' "$current_first_phase8")
+candidate_native=$(sed -n 's/^native_base_stores=//p' "$candidate_phase8")
+rollback_native=$(sed -n 's/^native_base_stores=//p' "$rollback_phase8")
+
+printf '%s\n' "$current_first_native" | tr ',' '\n' | grep "freebsd-native-kernel-$current_base_version$" >/dev/null || {
+  echo "current native store set missing expected kernel version" >&2
+  exit 1
+}
+printf '%s\n' "$candidate_native" | tr ',' '\n' | grep "freebsd-native-kernel-$candidate_base_version$" >/dev/null || {
+  echo "candidate native store set missing expected kernel version" >&2
+  exit 1
+}
+[ "$current_first_native" = "$rollback_native" ] || {
+  echo "rollback native store set did not return to the original current set" >&2
+  exit 1
+}
+
+cat >"$metadata_file" <<EOF
+workdir=$workdir
+current_first_metadata=$current_first_metadata
+candidate_metadata=$candidate_metadata
+rollback_metadata=$rollback_metadata
+current_first_closure=$current_first_closure
+candidate_closure=$candidate_closure
+rollback_closure=$rollback_closure
+current_base_version_label=$current_first_version
+candidate_base_version_label=$candidate_version
+rollback_base_version_label=$rollback_version
+current_native_base_stores=$current_first_native
+candidate_native_base_stores=$candidate_native
+rollback_native_base_stores=$rollback_native
+disk_capacity=$disk_capacity
+root_size=$root_size
+boot_backend=qemu-uefi-tcg
+base_rollforward_and_rollback=ok
+EOF
+
+if [ -n "$metadata_target" ]; then
+  mkdir -p "$(dirname "$metadata_target")"
+  cp "$metadata_file" "$metadata_target"
+fi
+
+printf 'PASS phase15-base-rollback-qemu\n'
+printf 'Work directory: %s\n' "$workdir"
+printf 'Metadata file: %s\n' "$metadata_file"
+if [ -n "$metadata_target" ]; then
+  printf 'Copied metadata to: %s\n' "$metadata_target"
+fi
+printf '%s\n' '--- metadata ---'
+cat "$metadata_file"
--- a/tests/system/run-phase15-base-rollback-xcpng.sh
+++ b/tests/system/run-phase15-base-rollback-xcpng.sh
@@ -0,0 +1,152 @@
+#!/bin/sh
+set -eu
+
+repo_root=${PROJECT_ROOT:-$(pwd)}
+script_dir=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
+os_template=${OS_TEMPLATE:-$script_dir/phase15-declarative-base-pid1-operating-system.scm.in}
+system_name=${SYSTEM_NAME:-phase15-operating-system}
+root_size=${ROOT_SIZE:-6g}
+current_base_name=${CURRENT_BASE_NAME:-stable-default}
+current_base_version=${CURRENT_BASE_VERSION:-15.0-STABLE}
+current_base_release=${CURRENT_BASE_RELEASE:-15.0-STABLE}
+current_base_branch=${CURRENT_BASE_BRANCH:-stable/15}
+candidate_base_name=${CANDIDATE_BASE_NAME:-stable-canary}
+candidate_base_version=${CANDIDATE_BASE_VERSION:-15.0-STABLE-p1}
+candidate_base_release=${CANDIDATE_BASE_RELEASE:-15.0-STABLE}
+candidate_base_branch=${CANDIDATE_BASE_BRANCH:-stable/15}
+metadata_target=${METADATA_OUT:-}
+cleanup=0
+
+if [ -n "${WORKDIR:-}" ]; then
+  workdir=$WORKDIR
+  mkdir -p "$workdir"
+else
+  workdir=$(mktemp -d /tmp/fruix-phase15-base-rollback-xcpng.XXXXXX)
+  cleanup=1
+fi
+if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then
+  cleanup=0
+fi
+
+metadata_file=$workdir/phase15-base-rollback-xcpng-metadata.txt
+current_template=$workdir/current-template.scm.in
+candidate_template=$workdir/candidate-template.scm.in
+
+cleanup_workdir() {
+  if [ "$cleanup" -eq 1 ]; then
+    rm -rf "$workdir"
+  fi
+}
+trap cleanup_workdir EXIT INT TERM
+
+render_template() {
+  output=$1
+  base_name=$2
+  base_version=$3
+  base_release=$4
+  base_branch=$5
+  sed \
+    -e "s|__BASE_NAME__|$base_name|g" \
+    -e "s|__BASE_VERSION_LABEL__|$base_version|g" \
+    -e "s|__BASE_RELEASE__|$base_release|g" \
+    -e "s|__BASE_BRANCH__|$base_branch|g" \
+    "$os_template" > "$output"
+}
+
+render_template "$current_template" "$current_base_name" "$current_base_version" "$current_base_release" "$current_base_branch"
+render_template "$candidate_template" "$candidate_base_name" "$candidate_base_version" "$candidate_base_release" "$candidate_base_branch"
+
+run_boot() {
+  name=$1
+  template=$2
+  metadata_out=$3
+  KEEP_WORKDIR=1 WORKDIR="$workdir/$name" METADATA_OUT="$metadata_out" \
+    OS_TEMPLATE="$template" SYSTEM_NAME="$system_name" ROOT_SIZE="$root_size" \
+    "$repo_root/tests/system/run-phase11-shepherd-pid1-xcpng.sh" >/dev/null
+}
+
+candidate_metadata=$workdir/candidate-metadata.txt
+rollback_metadata=$workdir/rollback-metadata.txt
+
+run_boot candidate "$candidate_template" "$candidate_metadata"
+run_boot rollback "$current_template" "$rollback_metadata"
+
+candidate_phase8=$(sed -n 's/^phase8_metadata=//p' "$candidate_metadata")
+rollback_phase8=$(sed -n 's/^phase8_metadata=//p' "$rollback_metadata")
+
+candidate_closure=$(sed -n 's/^closure_path=//p' "$candidate_metadata")
+rollback_closure=$(sed -n 's/^closure_path=//p' "$rollback_metadata")
+candidate_guest_ip=$(sed -n 's/^guest_ip=//p' "$candidate_metadata")
+rollback_guest_ip=$(sed -n 's/^guest_ip=//p' "$rollback_metadata")
+vm_id=$(sed -n 's/^vm_id=//p' "$rollback_metadata")
+vdi_id=$(sed -n 's/^vdi_id=//p' "$rollback_metadata")
+candidate_version=$(sed -n 's/^freebsd_base_version_label=//p' "$candidate_phase8")
+rollback_version=$(sed -n 's/^freebsd_base_version_label=//p' "$rollback_phase8")
+candidate_shepherd_pid=$(sed -n 's/^shepherd_pid=//p' "$candidate_metadata")
+rollback_shepherd_pid=$(sed -n 's/^shepherd_pid=//p' "$rollback_metadata")
+candidate_sshd=$(sed -n 's/^sshd_status=//p' "$candidate_metadata")
+rollback_sshd=$(sed -n 's/^sshd_status=//p' "$rollback_metadata")
+candidate_compat=$(sed -n 's/^compat_prefix_shims=//p' "$candidate_metadata")
+rollback_compat=$(sed -n 's/^compat_prefix_shims=//p' "$rollback_metadata")
+candidate_smoke=$(sed -n 's/^guile_module_smoke=//p' "$candidate_metadata")
+rollback_smoke=$(sed -n 's/^guile_module_smoke=//p' "$rollback_metadata")
+
+[ "$candidate_version" = "$candidate_base_version" ] || { echo "unexpected candidate version label" >&2; exit 1; }
+[ "$rollback_version" = "$current_base_version" ] || { echo "unexpected rollback version label" >&2; exit 1; }
+[ "$candidate_closure" != "$rollback_closure" ] || { echo "candidate and rollback closures unexpectedly match" >&2; exit 1; }
+for value in "$candidate_shepherd_pid" "$rollback_shepherd_pid"; do
+  [ "$value" = 1 ] || { echo "shepherd was not PID 1" >&2; exit 1; }
+done
+for value in "$candidate_sshd" "$rollback_sshd"; do
+  [ "$value" = running ] || { echo "sshd is not running during XCP-ng rollback validation" >&2; exit 1; }
+done
+for value in "$candidate_compat" "$rollback_compat"; do
+  [ "$value" = absent ] || { echo "compatibility prefix shims reappeared" >&2; exit 1; }
+done
+for value in "$candidate_smoke" "$rollback_smoke"; do
+  [ "$value" = ok ] || { echo "guest Guile module smoke failed" >&2; exit 1; }
+done
+
+candidate_native=$(sed -n 's/^native_base_stores=//p' "$candidate_phase8")
+rollback_native=$(sed -n 's/^native_base_stores=//p' "$rollback_phase8")
+printf '%s\n' "$candidate_native" | tr ',' '\n' | grep "freebsd-native-kernel-$candidate_base_version$" >/dev/null || {
+  echo "candidate native store set missing expected kernel version" >&2
+  exit 1
+}
+printf '%s\n' "$rollback_native" | tr ',' '\n' | grep "freebsd-native-kernel-$current_base_version$" >/dev/null || {
+  echo "rollback native store set missing expected kernel version" >&2
+  exit 1
+}
+
+cat >"$metadata_file" <<EOF
+workdir=$workdir
+candidate_metadata=$candidate_metadata
+rollback_metadata=$rollback_metadata
+candidate_closure=$candidate_closure
+rollback_closure=$rollback_closure
+candidate_base_version_label=$candidate_version
+rollback_base_version_label=$rollback_version
+candidate_native_base_stores=$candidate_native
+rollback_native_base_stores=$rollback_native
+guest_ip_candidate=$candidate_guest_ip
+guest_ip_rollback=$rollback_guest_ip
+vm_id=$vm_id
+vdi_id=$vdi_id
+root_size=$root_size
+boot_backend=xcp-ng-xo-cli
+base_rollforward_and_rollback=ok
+EOF
+
+if [ -n "$metadata_target" ]; then
+  mkdir -p "$(dirname "$metadata_target")"
+  cp "$metadata_file" "$metadata_target"
+fi
+
+printf 'PASS phase15-base-rollback-xcpng\n'
+printf 'Work directory: %s\n' "$workdir"
+printf 'Metadata file: %s\n' "$metadata_file"
+if [ -n "$metadata_target" ]; then
+  printf 'Copied metadata to: %s\n' "$metadata_target"
+fi
+printf '%s\n' '--- metadata ---'
+cat "$metadata_file"
--- a/tests/system/run-phase8-system-image.sh
+++ b/tests/system/run-phase8-system-image.sh
@@ -83,6 +83,15 @@ native_base_store_count=$(sed -n 's/^native_base_store_count=//p' "$build_metada
 native_base_stores=$(sed -n 's/^native_base_stores=//p' "$build_metadata")
 fruix_runtime_store_count=$(sed -n 's/^fruix_runtime_store_count=//p' "$build_metadata")
 fruix_runtime_stores=$(sed -n 's/^fruix_runtime_stores=//p' "$build_metadata")
+freebsd_base_name=$(sed -n 's/^freebsd_base_name=//p' "$build_metadata")
+freebsd_base_version_label=$(sed -n 's/^freebsd_base_version_label=//p' "$build_metadata")
+freebsd_base_release=$(sed -n 's/^freebsd_base_release=//p' "$build_metadata")
+freebsd_base_branch=$(sed -n 's/^freebsd_base_branch=//p' "$build_metadata")
+freebsd_base_source_root=$(sed -n 's/^freebsd_base_source_root=//p' "$build_metadata")
+freebsd_base_target=$(sed -n 's/^freebsd_base_target=//p' "$build_metadata")
+freebsd_base_target_arch=$(sed -n 's/^freebsd_base_target_arch=//p' "$build_metadata")
+freebsd_base_kernconf=$(sed -n 's/^freebsd_base_kernconf=//p' "$build_metadata")
+freebsd_base_file=$(sed -n 's/^freebsd_base_file=//p' "$build_metadata")
 host_base_provenance_file=$(sed -n 's/^host_base_provenance_file=//p' "$build_metadata")
 store_layout_file=$(sed -n 's/^store_layout_file=//p' "$build_metadata")
 host_freebsd_version=$(sed -n 's/^host_freebsd_version=//p' "$build_metadata")
@@ -158,6 +167,15 @@ native_base_store_count=$native_base_store_count
 native_base_stores=$native_base_stores
 fruix_runtime_store_count=$fruix_runtime_store_count
 fruix_runtime_stores=$fruix_runtime_stores
+freebsd_base_name=$freebsd_base_name
+freebsd_base_version_label=$freebsd_base_version_label
+freebsd_base_release=$freebsd_base_release
+freebsd_base_branch=$freebsd_base_branch
+freebsd_base_source_root=$freebsd_base_source_root
+freebsd_base_target=$freebsd_base_target
+freebsd_base_target_arch=$freebsd_base_target_arch
+freebsd_base_kernconf=$freebsd_base_kernconf
+freebsd_base_file=$freebsd_base_file
 host_base_provenance_file=$host_base_provenance_file
 store_layout_file=$store_layout_file
 host_freebsd_version=$host_freebsd_version