diff --git a/docs/GUIX_DIFFERENCES.md b/docs/GUIX_DIFFERENCES.md index 8118dd8..22b0ea8 100644 --- a/docs/GUIX_DIFFERENCES.md +++ b/docs/GUIX_DIFFERENCES.md @@ -263,6 +263,17 @@ The intent is: Compared with Guix, this is conceptually similar to keeping development-oriented state separate from the main runtime identity, but Fruix currently expresses it as a system-attached development overlay rather than through Guix's broader profile/tooling model. +Fruix now also has a narrow guest self-hosted native-build prototype helper at: + +- `/usr/local/bin/fruix-self-hosted-native-build` + +That helper does **not** just reuse the whole exported development shell wholesale. The validated prototype had to sanitize development-oriented variables such as `MAKEFLAGS`, `CPPFLAGS`, `CFLAGS`, `CXXFLAGS`, and `LDFLAGS` before `buildworld`, because those are convenient for smaller development tasks but can poison the FreeBSD world/kernel bootstrap path. + +The practical Fruix takeaway is: + +- the development overlay makes native base work possible inside the system +- but real guest self-hosted base builds still need their own stricter build contract + ## Where Fruix is intentionally trying to improve on Guix's representation Fruix is not trying to improve on Guix's core semantics. Guix already got those right. diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 4dcc661..4d6170a 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -42,6 +42,10 @@ Fruix currently has: - real XCP-ng boot of a development-enabled Fruix system - in-guest `buildworld` / `buildkernel` - staged `installworld` / `distribution` / `installkernel` +- a validated controlled guest self-hosted native base-build prototype via: + - `/usr/local/bin/fruix-self-hosted-native-build` + - result roots under `/var/lib/fruix/native-builds` + - in-guest source recovery from current-system metadata Validated boot modes still are: @@ -54,33 +58,32 @@ The validated Phase 18 installation work currently uses: ## Latest completed achievement -### 2026-04-05 — Phase 20.2 completed +### 2026-04-05 — Phase 20.3 completed -Fruix now has a validated intermediate path where the host still orchestrates the workflow, but real FreeBSD native base-build work runs inside a booted Fruix-managed FreeBSD guest. +Fruix now has a validated first controlled guest self-hosted native base-build prototype, on top of the already validated host-initiated in-guest build path. Highlights: -- development-enabled systems now expose canonical native-build compatibility links at: - - `/usr/include -> /run/current-system/development-profile/usr/include` - - `/usr/share/mk -> /run/current-system/development-profile/usr/share/mk` -- media builder versions were bumped so booted images and future installed targets pick up that rootfs layout change -- the validated guest build path now runs real FreeBSD native build steps inside the Fruix-managed guest: - - `buildworld` - - `buildkernel` - - `installworld` - - `distribution` - - `installkernel` -- staged install steps use: - - `DB_FROM_SRC=yes` - - so the staged install is driven by the declared source tree's account database rather than by the guest's minimal local `/etc` state -- the validated result now includes staged native artifact outputs for: +- development-enabled systems now ship an in-guest helper at: + - `/usr/local/bin/fruix-self-hosted-native-build` +- the helper recovers the declared materialized FreeBSD source store from: + - `/run/current-system/metadata/store-layout.scm` +- the helper records self-hosted build results under: + - `/var/lib/fruix/native-builds/` + - `/var/lib/fruix/native-builds/latest` +- heavy object/stage work remains under: + - `/var/tmp/fruix-self-hosted-native-builds/` +- the prototype exposed an important contract detail: + - a naive reuse of the development-shell exports polluted `buildworld` + - the validated helper therefore sanitizes development-oriented variables such as `MAKEFLAGS`, `CPPFLAGS`, `CFLAGS`, `CXXFLAGS`, and `LDFLAGS` before world/kernel bootstrap +- the validated result includes guest-recorded native artifact outputs for: - kernel - bootloader slice - headers / `usr/share/mk` Validation: -- `PASS phase20-host-initiated-native-build-xcpng` +- `PASS phase20-self-hosted-native-build-xcpng` Reports: @@ -88,6 +91,7 @@ Reports: - `docs/GUIX_DIFFERENCES.md` - `docs/reports/phase20-development-environment-freebsd.md` - `docs/reports/phase20-host-initiated-native-builds-freebsd.md` +- `docs/reports/phase20-self-hosted-native-builds-freebsd.md` ## Recent major milestones @@ -111,8 +115,11 @@ Reports: ## Next step -Per `docs/PLAN_4.md`, the next planned step is: +`docs/PLAN_4.md` currently ends at Phase 20.3, and that milestone sequence is now complete. -- **Phase 20.3** — reassess and potentially prototype guest self-hosted base builds +The next practical follow-up is therefore no longer another planned `PLAN_4` phase, but a product decision: -Phase 20.2 is now complete: Fruix validates host-initiated native FreeBSD base builds running inside the approved real XCP-ng Fruix guest path. +- whether to keep the new guest self-hosted helper as a narrow prototype while Phase 20.2 remains the default operator path +- or whether to invest in a broader guest-driven Fruix-native native-build workflow + +Phase 20.3 is now complete: Fruix validates a first controlled guest self-hosted native FreeBSD base-build prototype on the approved real XCP-ng path. diff --git a/docs/reports/phase20-self-hosted-native-builds-freebsd.md b/docs/reports/phase20-self-hosted-native-builds-freebsd.md new file mode 100644 index 0000000..f9fc35a --- /dev/null +++ b/docs/reports/phase20-self-hosted-native-builds-freebsd.md @@ -0,0 +1,201 @@ +# Phase 20.3: controlled guest self-hosted native base-build prototype + +Date: 2026-04-05 + +## Goal + +Reassess guest self-hosting now that Fruix has already completed the earlier source, installation, generation-layout, rollback, development-overlay, and host-initiated in-guest native-build steps. + +Phase 20.3 asked for real evidence about: + +- what self-hosting would improve +- what it would cost in complexity +- how it fits with the Fruix source/deployment model already in place + +## What changed + +### New in-guest helper + +Development-enabled systems now also ship: + +- `/usr/local/bin/fruix-self-hosted-native-build` + +This helper performs a controlled in-guest native FreeBSD base build using the system's own declared materialized source store recorded in: + +- `/run/current-system/metadata/store-layout.scm` + +The helper: + +1. verifies the development overlay is present +2. verifies the canonical compatibility links exist: + - `/usr/include` + - `/usr/share/mk` +3. recovers the materialized FreeBSD source store from current-system metadata +4. runs: + - `buildworld` + - `buildkernel` + - `installworld` + - `distribution` + - `installkernel` +5. stages narrower artifact outputs under: + - `/var/lib/fruix/native-builds//artifacts/` +6. records metadata and status under: + - `/var/lib/fruix/native-builds//` + - `/var/lib/fruix/native-builds/latest` + +The heavy object/stage work stays under: + +- `/var/tmp/fruix-self-hosted-native-builds/` + +so the installed-system result area remains smaller and more legible. + +### Important environment fix discovered during prototyping + +The first prototype attempt failed even though Phase 20.2 had already succeeded. + +Cause: + +- directly evaluating `fruix-development-environment` before `buildworld` exported development-oriented variables like: + - `MAKEFLAGS` + - `CPPFLAGS` + - `CFLAGS` + - `CXXFLAGS` + - `LDFLAGS` +- those are appropriate for smaller development builds, but they polluted FreeBSD's world/kernel bootstrap environment and broke the LLVM bootstrap phase + +Representative failure: + +- missing generated LLVM config headers during bootstrap (`llvm/Config/abi-breaking.h`) + +The validated fix was to make the self-hosted helper explicitly sanitize that environment first: + +- reset `PATH` to the normal base paths +- unset development-shell variables such as: + - `MAKEFLAGS` + - `CC`, `CXX`, `AR`, `RANLIB`, `NM` + - `CPPFLAGS`, `CFLAGS`, `CXXFLAGS`, `LDFLAGS` + - `FRUIX_DEVELOPMENT_*` + - `FRUIX_*` tool variables + +So the final 20.3 result is not “just reuse the development shell wholesale”. + +It is more precise: + +- use the development overlay for canonical paths and available content +- but run the real base-build steps in a cleaner, purpose-built helper environment + +### Closure invalidation + +To ensure the updated helper actually affects generated system closures, the operating-system closure spec now also records helper-version markers for development-enabled systems. + +That ensures guest images pick up helper changes instead of silently reusing an older cached closure path. + +### Validation harness + +Added: + +- `tests/system/run-phase20-self-hosted-native-build-xcpng.sh` + +This harness: + +1. boots the validated development-enabled Fruix guest on the approved XCP-ng path +2. verifies the new helper exists in the guest +3. invokes the helper from inside the guest +4. verifies the recorded result/status/`latest` pointer +5. validates the resulting staged artifact metadata and hashes + +## Validation + +Passing run: + +- `PASS phase20-self-hosted-native-build-xcpng` +- workdir: `/tmp/fruix-phase20-self-hosted-native-build-xcpng` + +Validated on the approved real XCP-ng path: + +- VM `90490f2e-e8fc-4b7a-388e-5c26f0157289` +- VDI `0f1f90d3-48ca-4fa2-91d8-fc6339b95743` + +Representative metadata: + +```text +run_id=20260405T150359Z +helper_version=2 +build_jobs=8 +source_store=/frx/store/12d7704362e95afc2697db63f168b878e082b372-freebsd-source-default +build_root=/var/tmp/fruix-self-hosted-native-builds/20260405T150359Z +result_root=/var/lib/fruix/native-builds/20260405T150359Z +latest_link=/var/lib/fruix/native-builds/latest +latest_target=/var/lib/fruix/native-builds/20260405T150359Z +status_value=ok +build_root_size=7.5G +result_root_size=343M +kernel_artifact_size=158M +headers_artifact_size=32M +bootloader_artifact_size=1.3M +sha_kernel=16950f116a52134b98e2f8e0dacc556e18fe254e4a0ac2c1741422dde281a341 +sha_loader=ea417846167ece270ada611624dca622ca38bd30125b9a125cd8ebb8b3600313 +sha_param=9eb140ca7d9666f3d484a4174c9acd94b45427db6292b4e17de19af2c6aa5219 +self_hosted_native_build=ok +``` + +Validated facts: + +- the development-enabled Fruix guest can now run a controlled self-hosted native base-build helper from inside the installed system itself +- the helper can recover the declared source store from current-system metadata without host-side parsing +- `buildworld` and `buildkernel` succeed in the guest +- staged `installworld`, `distribution`, and `installkernel` succeed in the guest +- the helper records a stable result directory and `latest` pointer under: + - `/var/lib/fruix/native-builds` +- the resulting artifact hashes match the earlier validated Phase 20.2 host-initiated in-guest path + +## What self-hosting improved + +The prototype demonstrates a few real improvements: + +- the build recipe itself now lives inside the Fruix-managed system, not only in a host-side SSH harness +- the guest can derive its own declared source input from current-system metadata +- result/state recording now has a Fruix-native installed-system location: + - `/var/lib/fruix/native-builds` +- the host no longer needs to spell out every `make` phase just to validate the in-guest path + +## What it cost in complexity + +The prototype also made the extra complexity visible: + +- the guest helper needs its own controlled environment contract +- a naive reuse of the development-shell exports was wrong for real `buildworld` +- helper-version invalidation had to be made explicit so closure caching would not hide helper changes +- the in-guest result/staging model now needs its own operator-facing conventions + +So the experiment did not eliminate complexity. + +It mostly moved some of it from the host harness into an explicit in-guest helper contract. + +## Decision after the prototype + +Phase 20.3 is complete because Fruix now has a **first controlled guest self-hosted native base-build prototype**. + +However, the evidence does **not** suggest replacing the Phase 20.2 path as the default operator workflow yet. + +The current recommendation is: + +- keep the **host-initiated in-guest native-build path** as the simpler default validation and orchestration flow +- keep the new **self-hosted helper** as a controlled prototype and stepping stone toward deeper guest-driven workflows + +That fits the existing Fruix model well: + +- source identity still comes from declared store-backed metadata +- deployment identity still comes from immutable closures under `/frx/store` +- the guest-side prototype adds a narrower in-system build/result workflow without replacing the existing deployment story + +## Result + +Phase 20.3 is complete. + +Fruix now has: + +- a validated host-orchestrated in-guest native base-build workflow +- and a validated first controlled guest self-hosted native base-build prototype + +That answers the Phase 20.3 question with real evidence instead of only prior caution. diff --git a/docs/system-deployment-workflow.md b/docs/system-deployment-workflow.md index f332cc8..f6b3aff 100644 --- a/docs/system-deployment-workflow.md +++ b/docs/system-deployment-workflow.md @@ -259,6 +259,41 @@ This is the current Phase 20.2 answer to “where should native base builds run? - **inside** a Fruix-managed FreeBSD environment - but still with the **host** driving the outer orchestration loop +### Controlled guest self-hosted native-build prototype + +Fruix now also has a narrower in-guest prototype helper at: + +- `/usr/local/bin/fruix-self-hosted-native-build` + +Intended use: + +```sh +FRUIX_SELF_HOSTED_NATIVE_BUILD_JOBS=8 \ + /usr/local/bin/fruix-self-hosted-native-build +``` + +That helper: + +1. verifies the development overlay and canonical compatibility links +2. recovers the materialized FreeBSD source store from: + - `/run/current-system/metadata/store-layout.scm` +3. runs the native FreeBSD build/install phases inside the guest +4. records results under: + - `/var/lib/fruix/native-builds/` + - `/var/lib/fruix/native-builds/latest` +5. keeps the heavier object/stage work under: + - `/var/tmp/fruix-self-hosted-native-builds/` + +Important current detail: + +- the self-hosted helper intentionally **sanitizes** development-shell exports such as `MAKEFLAGS`, `CPPFLAGS`, `CFLAGS`, `CXXFLAGS`, and `LDFLAGS` before `buildworld` +- directly reusing the full development-shell environment polluted FreeBSD's bootstrap path and was not reliable enough for real world/kernel builds + +So the validated Phase 20.3 answer is: + +- a controlled guest self-hosted base-build prototype now works +- but the simpler default operator flow should still be the Phase 20.2 host-initiated in-guest path unless there is a specific reason to push the build loop farther into the guest + ## Deployment patterns ### 1. Build-first workflow diff --git a/modules/fruix/system/freebsd/media.scm b/modules/fruix/system/freebsd/media.scm index cf27881..1342306 100644 --- a/modules/fruix/system/freebsd/media.scm +++ b/modules/fruix/system/freebsd/media.scm @@ -225,6 +225,8 @@ (chmod (string-append closure-path "/usr/local/bin/fruix") #o555)) (when (file-exists? (string-append closure-path "/usr/local/bin/fruix-development-environment")) (chmod (string-append closure-path "/usr/local/bin/fruix-development-environment") #o555)) + (when (file-exists? (string-append closure-path "/usr/local/bin/fruix-self-hosted-native-build")) + (chmod (string-append closure-path "/usr/local/bin/fruix-self-hosted-native-build") #o555)) (when (file-exists? (string-append closure-path "/boot/fruix-pid1")) (chmod (string-append closure-path "/boot/fruix-pid1") #o555)) (write-file (string-append closure-path "/parameters.scm") @@ -386,6 +388,9 @@ (when (file-exists? (string-append closure-path "/usr/local/bin/fruix-development-environment")) (symlink-force "/run/current-system/usr/local/bin/fruix-development-environment" (string-append rootfs "/usr/local/bin/fruix-development-environment"))) + (when (file-exists? (string-append closure-path "/usr/local/bin/fruix-self-hosted-native-build")) + (symlink-force "/run/current-system/usr/local/bin/fruix-self-hosted-native-build" + (string-append rootfs "/usr/local/bin/fruix-self-hosted-native-build"))) (symlink-force "/run/current-system/usr/local/etc/rc.d/fruix-activate" (string-append rootfs "/usr/local/etc/rc.d/fruix-activate")) (symlink-force "/run/current-system/usr/local/etc/rc.d/fruix-shepherd" diff --git a/modules/fruix/system/freebsd/model.scm b/modules/fruix/system/freebsd/model.scm index 331d041..1c15852 100644 --- a/modules/fruix/system/freebsd/model.scm +++ b/modules/fruix/system/freebsd/model.scm @@ -308,7 +308,8 @@ "usr/local/bin/fruix") (if (null? (operating-system-development-packages os)) '() - '("usr/local/bin/fruix-development-environment")) + '("usr/local/bin/fruix-development-environment" + "usr/local/bin/fruix-self-hosted-native-build")) (if (pid1-init-mode? os) '("boot/fruix-pid1") '()) @@ -330,6 +331,10 @@ (base-packages . ,(package-names (operating-system-base-packages os))) (development-package-count . ,(length (operating-system-development-packages os))) (development-packages . ,(package-names (operating-system-development-packages os))) + (development-environment-helper-version + . ,(if (null? (operating-system-development-packages os)) #f "1")) + (self-hosted-native-build-helper-version + . ,(if (null? (operating-system-development-packages os)) #f "2")) (user-count . ,(length (operating-system-users os))) (users . ,(map user-account-name (operating-system-users os))) (group-count . ,(length (operating-system-groups os))) diff --git a/modules/fruix/system/freebsd/render.scm b/modules/fruix/system/freebsd/render.scm index 7a443d9..8d52eda 100644 --- a/modules/fruix/system/freebsd/render.scm +++ b/modules/fruix/system/freebsd/render.scm @@ -785,6 +785,175 @@ "export MAKEFLAGS=\"-m $profile/usr/share/mk\"\n" "export PATH=\"/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$profile/bin:$profile/sbin:$profile/usr/bin:$profile/usr/sbin\"\n" "EOF\n")) +(define (render-self-hosted-native-build-script os) + (let* ((base-spec (freebsd-base-spec (operating-system-freebsd-base os))) + (target (assoc-ref base-spec 'target)) + (target-arch (assoc-ref base-spec 'target-arch)) + (kernconf (assoc-ref base-spec 'kernconf)) + (make-flags (or (assoc-ref base-spec 'make-flags) '())) + (build-common (string-join + (append (list (format #f "TARGET=~a" target) + (format #f "TARGET_ARCH=~a" target-arch) + (format #f "KERNCONF=~a" kernconf)) + make-flags) + " ")) + (install-common (string-append build-common " DB_FROM_SRC=yes"))) + (string-append + "#!/bin/sh\n" + "set -eu\n" + "umask 022\n" + "profile=/run/current-system/development-profile\n" + "[ -d \"$profile\" ] || {\n" + " echo \"fruix-self-hosted-native-build: development profile is not available\" >&2\n" + " exit 1\n" + "}\n" + "[ -x /usr/local/bin/fruix-development-environment ] || {\n" + " echo \"fruix-self-hosted-native-build: development environment helper is missing\" >&2\n" + " exit 1\n" + "}\n" + "PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin\n" + "unset MAKEOBJDIRPREFIX MAKEFLAGS CC CXX AR RANLIB NM CPPFLAGS CFLAGS CXXFLAGS LDFLAGS\n" + "unset FRUIX_DEVELOPMENT_PROFILE FRUIX_DEVELOPMENT_INCLUDE FRUIX_DEVELOPMENT_LIB FRUIX_DEVELOPMENT_SHARE_MK\n" + "unset FRUIX_DEVELOPMENT_BIN FRUIX_DEVELOPMENT_USR_BIN FRUIX_CC FRUIX_CXX FRUIX_AR FRUIX_RANLIB FRUIX_NM FRUIX_BMAKE\n" + "[ -L /usr/include ] || {\n" + " echo \"fruix-self-hosted-native-build: /usr/include compatibility link is missing\" >&2\n" + " exit 1\n" + "}\n" + "[ \"$(readlink /usr/include)\" = \"/run/current-system/development-profile/usr/include\" ] || {\n" + " echo \"fruix-self-hosted-native-build: /usr/include points at the wrong target\" >&2\n" + " exit 1\n" + "}\n" + "[ -L /usr/share/mk ] || {\n" + " echo \"fruix-self-hosted-native-build: /usr/share/mk compatibility link is missing\" >&2\n" + " exit 1\n" + "}\n" + "[ \"$(readlink /usr/share/mk)\" = \"/run/current-system/development-profile/usr/share/mk\" ] || {\n" + " echo \"fruix-self-hosted-native-build: /usr/share/mk points at the wrong target\" >&2\n" + " exit 1\n" + "}\n" + "jobs=${FRUIX_SELF_HOSTED_NATIVE_BUILD_JOBS:-$(sysctl -n hw.ncpu)}\n" + "case \"$jobs\" in\n" + " ''|*[!0-9]*)\n" + " echo \"fruix-self-hosted-native-build: invalid job count: $jobs\" >&2\n" + " exit 1\n" + " ;;\n" + "esac\n" + "run_id=${FRUIX_SELF_HOSTED_NATIVE_BUILD_ID:-$(date -u +%Y%m%dT%H%M%SZ)}\n" + "build_root_base=${FRUIX_SELF_HOSTED_NATIVE_BUILD_ROOT_BASE:-/var/tmp/fruix-self-hosted-native-builds}\n" + "result_root_base=${FRUIX_SELF_HOSTED_NATIVE_BUILD_OUTPUT_BASE:-/var/lib/fruix/native-builds}\n" + "build_root=$build_root_base/$run_id\n" + "result_root=$result_root_base/$run_id\n" + "logdir=$result_root/logs\n" + "status_file=$result_root/status\n" + "metadata_file=$result_root/metadata.txt\n" + "world_stage=$build_root/stage-world\n" + "kernel_stage=$build_root/stage-kernel\n" + "headers_artifact=$result_root/artifacts/headers\n" + "kernel_artifact=$result_root/artifacts/kernel\n" + "bootloader_artifact=$result_root/artifacts/bootloader\n" + "latest_link=$result_root_base/latest\n" + "mkdir -p \"$build_root\" \"$result_root\" \"$logdir\"\n" + "printf 'running\\n' > \"$status_file\"\n" + "fail_mark() {\n" + " rc=$?\n" + " if [ \"$rc\" -ne 0 ]; then\n" + " printf 'failed\\n' > \"$status_file\"\n" + " fi\n" + "}\n" + "trap fail_mark EXIT HUP INT TERM\n" + "closure=$(readlink /run/current-system)\n" + "store_layout=$closure/metadata/store-layout.scm\n" + "[ -f \"$store_layout\" ] || {\n" + " echo \"fruix-self-hosted-native-build: store layout metadata is missing\" >&2\n" + " exit 1\n" + "}\n" + "source_store=$(sed -n 's/.*\"\\(\\/frx\\/store\\/[^\"]*-freebsd-source-[^\"]*\\)\".*/\\1/p' \"$store_layout\" | head -n 1)\n" + "[ -n \"$source_store\" ] || {\n" + " echo \"fruix-self-hosted-native-build: failed to recover source store from store-layout.scm\" >&2\n" + " exit 1\n" + "}\n" + "source_root=$source_store/tree\n" + "[ -d \"$source_root\" ] || {\n" + " echo \"fruix-self-hosted-native-build: source root is missing: $source_root\" >&2\n" + " exit 1\n" + "}\n" + "mkdir -p \"$headers_artifact/usr\" \"$kernel_artifact/boot\" \"$bootloader_artifact/boot\"\n" + "export MAKEOBJDIRPREFIX=\"$build_root/obj\"\n" + "make -j\"$jobs\" -C \"$source_root\" " build-common " buildworld > \"$logdir/buildworld.log\" 2>&1\n" + "make -j\"$jobs\" -C \"$source_root\" " build-common " buildkernel > \"$logdir/buildkernel.log\" 2>&1\n" + "make -C \"$source_root\" " install-common " DESTDIR=\"$world_stage\" installworld > \"$logdir/installworld.log\" 2>&1\n" + "make -C \"$source_root\" " install-common " DESTDIR=\"$world_stage\" distribution > \"$logdir/distribution.log\" 2>&1\n" + "make -C \"$source_root\" " install-common " DESTDIR=\"$kernel_stage\" installkernel > \"$logdir/installkernel.log\" 2>&1\n" + "cp -a \"$kernel_stage/boot/kernel\" \"$kernel_artifact/boot/kernel\"\n" + "cp -a \"$world_stage/usr/include\" \"$headers_artifact/usr/include\"\n" + "mkdir -p \"$headers_artifact/usr/share\"\n" + "cp -a \"$world_stage/usr/share/mk\" \"$headers_artifact/usr/share/mk\"\n" + "cp -a \"$world_stage/boot/loader\" \"$bootloader_artifact/boot/loader\"\n" + "cp -a \"$world_stage/boot/loader.efi\" \"$bootloader_artifact/boot/loader.efi\"\n" + "cp -a \"$world_stage/boot/device.hints\" \"$bootloader_artifact/boot/device.hints\"\n" + "cp -a \"$world_stage/boot/defaults\" \"$bootloader_artifact/boot/defaults\"\n" + "cp -a \"$world_stage/boot/lua\" \"$bootloader_artifact/boot/lua\"\n" + "[ -f \"$kernel_artifact/boot/kernel/kernel\" ]\n" + "[ -f \"$headers_artifact/usr/include/sys/param.h\" ]\n" + "[ -f \"$headers_artifact/usr/share/mk/bsd.prog.mk\" ]\n" + "[ -f \"$bootloader_artifact/boot/loader.efi\" ]\n" + "[ -f \"$bootloader_artifact/boot/defaults/loader.conf\" ]\n" + "[ -f \"$bootloader_artifact/boot/lua/loader.lua\" ]\n" + "sha_kernel=$(sha256 -q \"$kernel_artifact/boot/kernel/kernel\")\n" + "sha_loader=$(sha256 -q \"$bootloader_artifact/boot/loader.efi\")\n" + "sha_param=$(sha256 -q \"$headers_artifact/usr/include/sys/param.h\")\n" + "buildworld_tail=$(tail -n 20 \"$logdir/buildworld.log\" | tr '\\n' ' ')\n" + "buildkernel_tail=$(tail -n 20 \"$logdir/buildkernel.log\" | tr '\\n' ' ')\n" + "installworld_tail=$(tail -n 20 \"$logdir/installworld.log\" | tr '\\n' ' ')\n" + "distribution_tail=$(tail -n 20 \"$logdir/distribution.log\" | tr '\\n' ' ')\n" + "installkernel_tail=$(tail -n 20 \"$logdir/installkernel.log\" | tr '\\n' ' ')\n" + "root_df=$(df -h / | tail -n 1 | tr -s ' ' | tr '\\t' ' ')\n" + "build_root_size=$(du -sh \"$build_root\" | awk '{print $1}')\n" + "result_root_size=$(du -sh \"$result_root\" | awk '{print $1}')\n" + "kernel_artifact_size=$(du -sh \"$kernel_artifact\" | awk '{print $1}')\n" + "headers_artifact_size=$(du -sh \"$headers_artifact\" | awk '{print $1}')\n" + "bootloader_artifact_size=$(du -sh \"$bootloader_artifact\" | awk '{print $1}')\n" + "rm -f \"$latest_link\"\n" + "ln -s \"$result_root\" \"$latest_link\"\n" + "cat >\"$metadata_file\" < \"$status_file\"\n" + "cat \"$metadata_file\"\n"))) (define* (operating-system-generated-files os #:key guile-store guile-extra-store shepherd-store) @@ -809,7 +978,9 @@ (if (null? (operating-system-development-packages os)) '() `(("usr/local/bin/fruix-development-environment" - . ,(render-development-environment-script os)))) + . ,(render-development-environment-script os)) + ("usr/local/bin/fruix-self-hosted-native-build" + . ,(render-self-hosted-native-build-script os)))) (if (pid1-init-mode? os) `(("boot/fruix-pid1" . ,(render-pid1-script os shepherd-store guile-store guile-extra-store))) '()) diff --git a/tests/system/run-phase20-self-hosted-native-build-xcpng.sh b/tests/system/run-phase20-self-hosted-native-build-xcpng.sh new file mode 100755 index 0000000..18bb2f1 --- /dev/null +++ b/tests/system/run-phase20-self-hosted-native-build-xcpng.sh @@ -0,0 +1,244 @@ +#!/bin/sh +set -eu + +repo_root=${PROJECT_ROOT:-$(pwd)} +os_template=${OS_TEMPLATE:-$repo_root/tests/system/phase20-development-operating-system.scm.in} +system_name=${SYSTEM_NAME:-phase20-operating-system} +root_size=${ROOT_SIZE:-20g} +metadata_target=${METADATA_OUT:-} +root_authorized_key_file=${ROOT_AUTHORIZED_KEY_FILE:-$HOME/.ssh/id_ed25519.pub} +root_ssh_private_key_file=${ROOT_SSH_PRIVATE_KEY_FILE:-$HOME/.ssh/id_ed25519} +cleanup=0 + +if [ -n "${WORKDIR:-}" ]; then + workdir=$WORKDIR + mkdir -p "$workdir" +else + workdir=$(mktemp -d /tmp/fruix-phase20-self-hosted-native-build-xcpng.XXXXXX) + cleanup=1 +fi +if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then + cleanup=0 +fi + +inner_metadata=$workdir/phase20-self-hosted-inner-metadata.txt +metadata_file=$workdir/phase20-self-hosted-native-build-xcpng-metadata.txt + +action_cleanup() { + if [ "$cleanup" -eq 1 ]; then + rm -rf "$workdir" + fi +} +trap action_cleanup EXIT INT TERM + +KEEP_WORKDIR=1 WORKDIR="$workdir/inner" METADATA_OUT="$inner_metadata" \ + ROOT_AUTHORIZED_KEY_FILE="$root_authorized_key_file" \ + ROOT_SSH_PRIVATE_KEY_FILE="$root_ssh_private_key_file" \ + OS_TEMPLATE="$os_template" SYSTEM_NAME="$system_name" ROOT_SIZE="$root_size" \ + "$repo_root/tests/system/run-phase20-development-environment-xcpng.sh" + +phase8_metadata=$(sed -n 's/^phase8_metadata=//p' "$inner_metadata") +closure_path=$(sed -n 's/^closure_path=//p' "$inner_metadata") +closure_base=$(sed -n 's/^closure_base=//p' "$inner_metadata") +guest_ip=$(sed -n 's/^guest_ip=//p' "$inner_metadata") +vm_id=$(sed -n 's/^vm_id=//p' "$inner_metadata") +vdi_id=$(sed -n 's/^vdi_id=//p' "$inner_metadata") +shepherd_pid=$(sed -n 's/^shepherd_pid=//p' "$inner_metadata") +sshd_status=$(sed -n 's/^sshd_status=//p' "$inner_metadata") +compat_prefix_shims=$(sed -n 's/^compat_prefix_shims=//p' "$inner_metadata") +guile_module_smoke=$(sed -n 's/^guile_module_smoke=//p' "$inner_metadata") + +[ "$shepherd_pid" = 1 ] || { echo "shepherd was not PID 1" >&2; exit 1; } +[ "$sshd_status" = running ] || { echo "sshd is not running" >&2; exit 1; } +[ "$compat_prefix_shims" = absent ] || { echo "compatibility prefix shims reappeared" >&2; exit 1; } +[ "$guile_module_smoke" = ok ] || { echo "guest Guile module smoke failed" >&2; exit 1; } + +ssh_guest() { + ssh -i "$root_ssh_private_key_file" \ + -o BatchMode=yes \ + -o StrictHostKeyChecking=no \ + -o UserKnownHostsFile=/dev/null \ + -o ConnectTimeout=5 \ + root@"$guest_ip" "$@" +} + +guest_build_jobs=${GUEST_BUILD_JOBS:-$(ssh_guest 'sysctl -n hw.ncpu')} +case "$guest_build_jobs" in + ''|*[!0-9]*) + echo "invalid guest build job count: $guest_build_jobs" >&2 + exit 1 + ;; +esac + +ssh_guest '[ -x /usr/local/bin/fruix-self-hosted-native-build ]' +ssh_guest '[ -L /usr/include ]' +ssh_guest '[ -L /usr/share/mk ]' + +self_hosted_metadata=$(ssh_guest env FRUIX_SELF_HOSTED_NATIVE_BUILD_JOBS="$guest_build_jobs" /usr/local/bin/fruix-self-hosted-native-build) + +run_id=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^run_id=//p') +helper_version=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^helper_version=//p') +source_store=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^source_store=//p') +source_root=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^source_root=//p') +build_jobs=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^build_jobs=//p') +build_common=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^build_common=//p') +install_common=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^install_common=//p') +build_root=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^build_root=//p') +result_root=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^result_root=//p') +logdir=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^logdir=//p') +status_file=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^status_file=//p') +guest_metadata_file=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^metadata_file=//p') +world_stage=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^world_stage=//p') +kernel_stage=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^kernel_stage=//p') +kernel_artifact=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^kernel_artifact=//p') +headers_artifact=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^headers_artifact=//p') +bootloader_artifact=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^bootloader_artifact=//p') +latest_link=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^latest_link=//p') +root_df=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^root_df=//p') +build_root_size=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^build_root_size=//p') +result_root_size=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^result_root_size=//p') +kernel_artifact_size=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^kernel_artifact_size=//p') +headers_artifact_size=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^headers_artifact_size=//p') +bootloader_artifact_size=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^bootloader_artifact_size=//p') +sha_kernel=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^sha_kernel=//p') +sha_loader=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^sha_loader=//p') +sha_param=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^sha_param=//p') +buildworld_tail=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^buildworld_tail=//p') +buildkernel_tail=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^buildkernel_tail=//p') +installworld_tail=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^installworld_tail=//p') +distribution_tail=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^distribution_tail=//p') +installkernel_tail=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^installkernel_tail=//p') +self_hosted_native_build=$(printf '%s\n' "$self_hosted_metadata" | sed -n 's/^self_hosted_native_build=//p') + +status_value=$(ssh_guest "cat '$status_file'") +latest_target=$(ssh_guest "readlink '$latest_link'") + +[ "$helper_version" = 2 ] || { echo "unexpected helper version: $helper_version" >&2; exit 1; } +[ "$build_jobs" = "$guest_build_jobs" ] || { echo "unexpected build job count: $build_jobs" >&2; exit 1; } +[ "$status_value" = ok ] || { echo "self-hosted build status is not ok: $status_value" >&2; exit 1; } +[ "$latest_target" = "$result_root" ] || { echo "latest link target mismatch: $latest_target" >&2; exit 1; } +[ "$self_hosted_native_build" = ok ] || { echo "self-hosted build marker missing" >&2; exit 1; } + +case "$source_store" in + /frx/store/*-freebsd-source-*) : ;; + *) echo "unexpected source store path: $source_store" >&2; exit 1 ;; +esac +case "$source_root" in + /frx/store/*-freebsd-source-*/*) : ;; + *) echo "unexpected source root path: $source_root" >&2; exit 1 ;; +esac +case "$build_root" in + /var/tmp/fruix-self-hosted-native-builds/*) : ;; + *) echo "unexpected build root: $build_root" >&2; exit 1 ;; +esac +case "$result_root" in + /var/lib/fruix/native-builds/*) : ;; + *) echo "unexpected result root: $result_root" >&2; exit 1 ;; +esac +case "$latest_link" in + /var/lib/fruix/native-builds/latest) : ;; + *) echo "unexpected latest link path: $latest_link" >&2; exit 1 ;; +esac +printf '%s\n' "$run_id" | grep -E '^[0-9]{8}T[0-9]{6}Z$' >/dev/null || { + echo "unexpected run id: $run_id" >&2 + exit 1 +} +printf '%s\n' "$sha_kernel" | grep -E '^[0-9a-f]{64}$' >/dev/null || { + echo "invalid kernel sha256: $sha_kernel" >&2 + exit 1 +} +printf '%s\n' "$sha_loader" | grep -E '^[0-9a-f]{64}$' >/dev/null || { + echo "invalid loader sha256: $sha_loader" >&2 + exit 1 +} +printf '%s\n' "$sha_param" | grep -E '^[0-9a-f]{64}$' >/dev/null || { + echo "invalid param.h sha256: $sha_param" >&2 + exit 1 +} +case "$buildworld_tail" in + *'World build completed on'*) : ;; + *) echo "buildworld log does not show completion" >&2; exit 1 ;; +esac +case "$buildkernel_tail" in + *'Kernel(s) GENERIC built in'*) : ;; + *) echo "buildkernel log does not show successful kernel build" >&2; exit 1 ;; +esac +case "$installworld_tail" in + *'Install world completed in'*) : ;; + *) echo "installworld log does not show successful completion" >&2; exit 1 ;; +esac +case "$installkernel_tail" in + *'Install kernel(s) GENERIC completed in'*) : ;; + *) echo "installkernel log does not show successful completion" >&2; exit 1 ;; +esac +printf '%s\n' "$build_common" | grep -F 'TARGET=amd64 TARGET_ARCH=amd64 KERNCONF=GENERIC' >/dev/null || { + echo "unexpected build_common flags: $build_common" >&2 + exit 1 +} +printf '%s\n' "$install_common" | grep -F 'DB_FROM_SRC=yes' >/dev/null || { + echo "install_common is missing DB_FROM_SRC=yes" >&2 + exit 1 +} + +cat >"$metadata_file" <