diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 07a9c73..068fff5 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -1,5 +1,53 @@ # Progress +## 2026-04-03 — Phase 17.2 completed: Fruix now boots systems from distinct declared FreeBSD source revisions + +Completed work: + +- added boot validation harness: + - `tests/system/run-phase17-source-revisions-qemu.sh` +- the new harness renders the Phase 17 Git and `src.txz` operating-system templates and boots both systems through the validated: + - QEMU + - UEFI + - TCG + - `shepherd-pid1` + path +- validated booted source identities: + - Git source: + - ref: `stable/15` + - commit: `332708a606f6bf0841c1d4a74c0d067f5640fe89` + - `src.txz` source: + - `https://download.freebsd.org/releases/amd64/15.0-RELEASE/src.txz` + - sha256: + - `83c3e8157b6d7afcae57167fda75693bf1e5f581ca149a6ecb2d398b71bdfab0` +- confirmed image/build metadata for both boots records: + - declared source kind + - ref/commit or archive URL/sha256 + - materialized source store path + - distinct native base store sets +- wrote: + - `docs/reports/phase17-source-revision-boots-freebsd.md` + +Validation: + +- `PASS phase17-source-revisions-qemu` +- validated distinct booted closures: + - Git: + - `/frx/store/d6cbcc76f57fa9c392a80fe20e7499f7a837aab4fb96ea056e624cde95bc70c8-fruix-system-fruix-freebsd` + - `src.txz`: + - `/frx/store/02268e19930facb32e12b6ec191f2e5704d1e81033baf3637a889ad15924ff88-fruix-system-fruix-freebsd` +- validated both guests reached the validated runtime state: + - Shepherd as PID 1 + - `sshd` running +- validated distinct materialized source stores and distinct native kernel/bootloader/runtime outputs for the two boots + +Current assessment: + +- Phase 17.2 is complete +- Fruix can now both build **and boot** systems from distinct declared FreeBSD source revisions +- the next step is Phase 17.3: + - document the intended policy for source provenance, caching, invalidation, and update semantics before installation work depends on it + ## 2026-04-03 — Phase 17.1 completed: side-by-side FreeBSD source revisions now coexist in `/frx/store` Completed work: diff --git a/docs/PROG_SUMMARY.md b/docs/PROG_SUMMARY.md index 1a4b810..44f3752 100644 --- a/docs/PROG_SUMMARY.md +++ b/docs/PROG_SUMMARY.md @@ -29,6 +29,7 @@ Completed milestones include: into `/frx/store`, with cache-backed provenance under `/frx/var/cache/fruix/freebsd-source`. - **Source-driven native base builds**: native FreeBSD kernel/bootloader/runtime artifacts now consume those materialized source snapshots rather than ambient `/usr/src`, and their build metadata records both the declared source and the effective materialized source identity. - **Side-by-side source revisions**: Fruix can now keep distinct FreeBSD source identities side by side in `/frx/store` and produce distinct native base outputs from them, even when the visible base version label is held constant. +- **Source-driven boot validation**: Fruix can now also boot systems built from distinct declared FreeBSD source revisions while preserving those source identities in image/build metadata. - **Base upgrade story**: Fruix can now keep distinct declared base versions side by side in `/frx/store` and roll forward / back between them through the normal system deployment flow. ## Major pain points now behind us @@ -43,7 +44,7 @@ Completed milestones include: ## Major pain points still ahead - **True store-native runtime artifacts**: some historical build/install prefixes are still embedded in binaries and metadata. They are no longer required at runtime, but the local Guile/guile-extra/Shepherd build/install flow should still be moved to a genuinely store-native prefix from the start. -- **Source-revision boot validation**: Fruix now has side-by-side source-driven native outputs, but it still needs stronger boot validation and update/rollback exercises across distinct fetched source revisions. +- **Source provenance/update policy**: Fruix now has side-by-side source-driven boots, but it still needs the repo-level policy for source caching, refresh, invalidation, and update semantics before later installation/deployment work depends on it. - **Boot-path simplification**: Fruix now supports both the legacy `freebsd-init+rc.d-shepherd` path and the more Guix-like `shepherd-pid1` path. We still need to decide whether Shepherd PID 1 becomes the preferred/default architecture. - **Reduce transitional FreeBSD glue**: more of the current bootstrap/activation/runtime setup should become cleaner and less prototype-specific over time. - **Tooling and platform constraints**: local bhyve remains blocked by missing nested virtualization under Xen, and XO permissions still prevent creating/importing new VDIs; current validation must keep reusing the approved VM/VDI path. diff --git a/docs/reports/phase17-source-revision-boots-freebsd.md b/docs/reports/phase17-source-revision-boots-freebsd.md new file mode 100644 index 0000000..7225f06 --- /dev/null +++ b/docs/reports/phase17-source-revision-boots-freebsd.md @@ -0,0 +1,98 @@ +# Phase 17.2: boot systems from distinct declared FreeBSD source revisions + +Date: 2026-04-03 + +## Goal + +Phase 17.2 extends Phase 17.1 from: + +- side-by-side source-driven builds + +to: + +- side-by-side source-driven **boots**. + +The important requirement was not visible runtime behavior differences between the guests. The requirement was that Fruix should be able to: + +- build bootable systems from at least two distinct declared source revisions +- boot both systems with the validated native base path +- preserve source identity in the resulting system/image metadata + +## Implementation + +Added boot validation harness: + +- `tests/system/run-phase17-source-revisions-qemu.sh` + +This script renders the Phase 17 source templates from Phase 17.1 and then boots two systems under the already-validated QEMU/UEFI/TCG path: + +- Git-backed source: + - ref: `stable/15` + - pinned commit: `332708a606f6bf0841c1d4a74c0d067f5640fe89` +- release archive source: + - `https://download.freebsd.org/releases/amd64/15.0-RELEASE/src.txz` + - sha256: + - `83c3e8157b6d7afcae57167fda75693bf1e5f581ca149a6ecb2d398b71bdfab0` + +The harness reuses: + +- `tests/system/run-phase11-shepherd-pid1-qemu.sh` + +and checks both guest runtime behavior and image/build provenance metadata. + +## Validation + +Passing run: + +- `PASS phase17-source-revisions-qemu` + +Confirmed booted systems from two distinct source identities: + +```text +git_closure=/frx/store/d6cbcc76f57fa9c392a80fe20e7499f7a837aab4fb96ea056e624cde95bc70c8-fruix-system-fruix-freebsd +txz_closure=/frx/store/02268e19930facb32e12b6ec191f2e5704d1e81033baf3637a889ad15924ff88-fruix-system-fruix-freebsd +``` + +Confirmed source metadata recorded in image/build artifacts: + +```text +git_source_kind=git +git_source_ref=stable/15 +git_source_commit=332708a606f6bf0841c1d4a74c0d067f5640fe89 +git_materialized_source_store=/frx/store/c9928605fa906b90a600dafeebe5005dd18ad3b8e62b7111d9d13ad60ee56490-freebsd-source-stable15-side-a + +txz_source_kind=src-txz +txz_source_url=https://download.freebsd.org/releases/amd64/15.0-RELEASE/src.txz +txz_source_sha256=83c3e8157b6d7afcae57167fda75693bf1e5f581ca149a6ecb2d398b71bdfab0 +txz_materialized_source_store=/frx/store/5eaeff5c6c55a95b6531d9cf2e1824cd4368d81c614608426bee1a5d2a664dc5-freebsd-source-release15-side-b +``` + +Confirmed distinct native base outputs used by the two boots: + +```text +git_native_base_stores=/frx/store/4b615431ec25c500a3bf0ed70ce39e2ebf4f584994a53756268e4383962bc86b-freebsd-native-kernel-15.0-source-side-by-side,/frx/store/3a5a0b2b88b4757cf9cb4e3040f992d8fdb5bd9a7f1b186da983854cd95392c5-freebsd-native-bootloader-15.0-source-side-by-side,/frx/store/177f78e7f2932986a380187eb09dc34cc2cd9a146c5ed1fe1f00aae15ddf78d9-freebsd-native-runtime-15.0-source-side-by-side + +txz_native_base_stores=/frx/store/0c5141a86fa9c1974102f2bd8766eb3ab787b97dcccb71f17d80aefbe8ed4f3e-freebsd-native-kernel-15.0-source-side-by-side,/frx/store/3de6592f50a735d8461662cb393fc413325ce24ded45d4bb494525896f8cb5eb-freebsd-native-bootloader-15.0-source-side-by-side,/frx/store/46d256305198ee7d745b9032c71085aba97d55fdf7a0d3d2017dd4455173205d-freebsd-native-runtime-15.0-source-side-by-side +``` + +Confirmed both guests booted successfully through the validated PID 1 path: + +- Shepherd ran as PID 1 in both boots +- `sshd` was running in both boots +- boot backend: + - `qemu-uefi-tcg` + +Validation artifacts: + +- Git serial log: + - `/tmp/fruix-phase17-source-qemu.7Za50q/git/serial.log` +- `src.txz` serial log: + - `/tmp/fruix-phase17-source-qemu.7Za50q/txz/serial.log` + +## Result + +Phase 17.2 is complete. + +Fruix now boots systems built from at least two distinct declared FreeBSD source revisions while preserving those source identities in system/image metadata. + +That means Phase 17 is no longer just about build-time coexistence. The validated native boot path now also works across distinct source identities. diff --git a/tests/system/run-phase17-source-revisions-qemu.sh b/tests/system/run-phase17-source-revisions-qemu.sh new file mode 100755 index 0000000..0e41199 --- /dev/null +++ b/tests/system/run-phase17-source-revisions-qemu.sh @@ -0,0 +1,194 @@ +#!/bin/sh +set -eu + +repo_root=${PROJECT_ROOT:-$(pwd)} +script_dir=$(CDPATH= cd -- "$(dirname "$0")" && pwd) +git_template=${GIT_TEMPLATE:-$script_dir/phase17-git-source-operating-system.scm.in} +txz_template=${TXZ_TEMPLATE:-$script_dir/phase17-txz-source-operating-system.scm.in} +system_name=${SYSTEM_NAME:-phase17-operating-system} +disk_capacity=${DISK_CAPACITY:-12g} +root_size=${ROOT_SIZE:-10g} +base_name=${BASE_NAME:-source-side-by-side} +base_version_label=${BASE_VERSION_LABEL:-15.0-source-side-by-side} +git_base_release=${GIT_BASE_RELEASE:-15.0-STABLE} +git_base_branch=${GIT_BASE_BRANCH:-stable/15} +git_source_name=${GIT_SOURCE_NAME:-stable15-side-a} +git_source_ref=${GIT_SOURCE_REF:-stable/15} +git_source_commit=${GIT_SOURCE_COMMIT:-332708a606f6bf0841c1d4a74c0d067f5640fe89} +git_declared_source_root=${GIT_DECLARED_SOURCE_ROOT:-/var/empty/fruix-unused-source-root-git} +txz_base_release=${TXZ_BASE_RELEASE:-15.0-RELEASE} +txz_base_branch=${TXZ_BASE_BRANCH:-releng/15.0} +txz_source_name=${TXZ_SOURCE_NAME:-release15-side-b} +txz_source_url=${TXZ_SOURCE_URL:-https://download.freebsd.org/releases/amd64/15.0-RELEASE/src.txz} +txz_source_sha256=${TXZ_SOURCE_SHA256:-83c3e8157b6d7afcae57167fda75693bf1e5f581ca149a6ecb2d398b71bdfab0} +txz_declared_source_root=${TXZ_DECLARED_SOURCE_ROOT:-/var/empty/fruix-unused-source-root-txz} +metadata_target=${METADATA_OUT:-} +cleanup=0 + +if [ -n "${WORKDIR:-}" ]; then + workdir=$WORKDIR + mkdir -p "$workdir" +else + workdir=$(mktemp -d /tmp/fruix-phase17-source-qemu.XXXXXX) + cleanup=1 +fi +if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then + cleanup=0 +fi + +metadata_file=$workdir/phase17-source-revisions-qemu-metadata.txt +git_rendered_template=$workdir/git-template.scm.in +txz_rendered_template=$workdir/txz-template.scm.in + +tidy() { + if [ "$cleanup" -eq 1 ]; then + rm -rf "$workdir" 2>/dev/null || sudo rm -rf "$workdir" + fi +} +trap tidy EXIT INT TERM + +[ -f "$git_template" ] || { echo "missing git template: $git_template" >&2; exit 1; } +[ -f "$txz_template" ] || { echo "missing txz template: $txz_template" >&2; exit 1; } + +sed \ + -e "s|__BASE_NAME__|$base_name|g" \ + -e "s|__BASE_VERSION_LABEL__|$base_version_label|g" \ + -e "s|__BASE_RELEASE__|$git_base_release|g" \ + -e "s|__BASE_BRANCH__|$git_base_branch|g" \ + -e "s|__SOURCE_NAME__|$git_source_name|g" \ + -e "s|__SOURCE_REF__|$git_source_ref|g" \ + -e "s|__SOURCE_COMMIT__|$git_source_commit|g" \ + -e "s|__DECLARED_SOURCE_ROOT__|$git_declared_source_root|g" \ + "$git_template" > "$git_rendered_template" + +sed \ + -e "s|__BASE_NAME__|$base_name|g" \ + -e "s|__BASE_VERSION_LABEL__|$base_version_label|g" \ + -e "s|__BASE_RELEASE__|$txz_base_release|g" \ + -e "s|__BASE_BRANCH__|$txz_base_branch|g" \ + -e "s|__SOURCE_NAME__|$txz_source_name|g" \ + -e "s|__SOURCE_URL__|$txz_source_url|g" \ + -e "s|__SOURCE_SHA256__|$txz_source_sha256|g" \ + -e "s|__DECLARED_SOURCE_ROOT__|$txz_declared_source_root|g" \ + "$txz_template" > "$txz_rendered_template" + +run_boot() { + name=$1 + template=$2 + metadata_out=$3 + qemu_port=$4 + KEEP_WORKDIR=1 WORKDIR="$workdir/$name" METADATA_OUT="$metadata_out" \ + OS_TEMPLATE="$template" SYSTEM_NAME="$system_name" DISK_CAPACITY="$disk_capacity" ROOT_SIZE="$root_size" \ + QEMU_SSH_PORT="$qemu_port" "$repo_root/tests/system/run-phase11-shepherd-pid1-qemu.sh" >/dev/null +} + +git_metadata=$workdir/git-metadata.txt +txz_metadata=$workdir/txz-metadata.txt +run_boot git "$git_rendered_template" "$git_metadata" 10022 +run_boot txz "$txz_rendered_template" "$txz_metadata" 10023 + +field() { + key=$1 + file=$2 + sed -n "s/^$key=//p" "$file" | tail -n 1 +} + +git_phase8=$(field phase8_metadata "$git_metadata") +txz_phase8=$(field phase8_metadata "$txz_metadata") + +[ -f "$git_phase8" ] || { echo "missing git phase8 metadata: $git_phase8" >&2; exit 1; } +[ -f "$txz_phase8" ] || { echo "missing txz phase8 metadata: $txz_phase8" >&2; exit 1; } + +git_phase8_build=$(field build_metadata "$git_phase8") +txz_phase8_build=$(field build_metadata "$txz_phase8") +[ -f "$git_phase8_build" ] || { echo "missing git phase8 build metadata: $git_phase8_build" >&2; exit 1; } +[ -f "$txz_phase8_build" ] || { echo "missing txz phase8 build metadata: $txz_phase8_build" >&2; exit 1; } + +git_closure=$(field closure_path "$git_metadata") +txz_closure=$(field closure_path "$txz_metadata") +git_shepherd_pid=$(field shepherd_pid "$git_metadata") +txz_shepherd_pid=$(field shepherd_pid "$txz_metadata") +git_sshd=$(field sshd_status "$git_metadata") +txz_sshd=$(field sshd_status "$txz_metadata") +git_serial_log=$(field serial_log "$git_metadata") +txz_serial_log=$(field serial_log "$txz_metadata") + +git_source_kind=$(field freebsd_source_kind "$git_phase8_build") +txz_source_kind=$(field freebsd_source_kind "$txz_phase8_build") +git_source_ref_out=$(field freebsd_source_ref "$git_phase8_build") +git_source_commit_out=$(field freebsd_source_commit "$git_phase8_build") +txz_source_url_out=$(field freebsd_source_url "$txz_phase8_build") +txz_source_sha256_out=$(field freebsd_source_sha256 "$txz_phase8_build") +git_materialized_count=$(field materialized_source_store_count "$git_phase8_build") +txz_materialized_count=$(field materialized_source_store_count "$txz_phase8_build") +git_materialized_store=$(field materialized_source_stores "$git_phase8_build") +txz_materialized_store=$(field materialized_source_stores "$txz_phase8_build") +git_native_stores=$(field native_base_stores "$git_phase8_build") +txz_native_stores=$(field native_base_stores "$txz_phase8_build") + +[ "$git_shepherd_pid" = 1 ] || { echo "git boot did not run Shepherd as PID 1" >&2; exit 1; } +[ "$txz_shepherd_pid" = 1 ] || { echo "txz boot did not run Shepherd as PID 1" >&2; exit 1; } +[ "$git_sshd" = running ] || { echo "git boot does not have sshd running" >&2; exit 1; } +[ "$txz_sshd" = running ] || { echo "txz boot does not have sshd running" >&2; exit 1; } +[ "$git_closure" != "$txz_closure" ] || { echo "git and txz boot closures unexpectedly match" >&2; exit 1; } +[ "$git_source_kind" = git ] || { echo "unexpected git boot source kind: $git_source_kind" >&2; exit 1; } +[ "$txz_source_kind" = src-txz ] || { echo "unexpected txz boot source kind: $txz_source_kind" >&2; exit 1; } +[ "$git_source_ref_out" = "$git_source_ref" ] || { echo "unexpected git boot source ref: $git_source_ref_out" >&2; exit 1; } +[ "$git_source_commit_out" = "$git_source_commit" ] || { echo "unexpected git boot source commit: $git_source_commit_out" >&2; exit 1; } +[ "$txz_source_url_out" = "$txz_source_url" ] || { echo "unexpected txz boot source URL: $txz_source_url_out" >&2; exit 1; } +[ "$txz_source_sha256_out" = "$txz_source_sha256" ] || { echo "unexpected txz boot source sha256: $txz_source_sha256_out" >&2; exit 1; } +[ "$git_materialized_count" = 1 ] || { echo "unexpected git materialized source count: $git_materialized_count" >&2; exit 1; } +[ "$txz_materialized_count" = 1 ] || { echo "unexpected txz materialized source count: $txz_materialized_count" >&2; exit 1; } +[ "$git_materialized_store" != "$txz_materialized_store" ] || { echo "git and txz materialized source stores unexpectedly match" >&2; exit 1; } +[ "$git_native_stores" != "$txz_native_stores" ] || { echo "git and txz native base stores unexpectedly match" >&2; exit 1; } + +case "$git_materialized_store" in + /frx/store/*-freebsd-source-$git_source_name) : ;; + *) echo "unexpected git materialized source store: $git_materialized_store" >&2; exit 1 ;; +esac +case "$txz_materialized_store" in + /frx/store/*-freebsd-source-$txz_source_name) : ;; + *) echo "unexpected txz materialized source store: $txz_materialized_store" >&2; exit 1 ;; +esac + +cat >"$metadata_file" <