Build native bases from materialized FreeBSD sources

docs/PROGRESS.md

@@ -1,5 +1,64 @@
 # Progress
 
+## 2026-04-03 — Phase 16.3 completed: native FreeBSD base builds now consume materialized source inputs
+
+Completed work:
+
+- refactored native FreeBSD package materialization in `modules/fruix/system/freebsd.scm` so native packages now:
+  - reconstruct the declared `freebsd-source` from the package plan
+  - materialize that source under Fruix control
+  - rewrite the native build plan to use the materialized source root
+  - add the materialized source store path to package references
+- native build manifests now include:
+  - `declared-source`
+  - `materialized-source`
+- native `.freebsd-native-build-info.scm` now records:
+  - declared source
+  - materialized source store path
+  - materialized source root
+  - materialized source tree sha256
+  - effective source details such as resolved Git commit / archive sha256
+- native build common manifests now reuse the materialized source tree hash when available
+- package materialization caching is now keyed by manifest identity instead of only package name/version, so distinct source-driven variants do not collide in-process
+- system closure materialization now keeps a dedicated source-materialization cache and records:
+  - `metadata/freebsd-source-materializations.scm`
+  - `materialized-source-store-count`
+  - `materialized-source-stores`
+- system closure references now include the materialized source stores explicitly
+- `scripts/fruix.scm` now emits for `fruix system build` and `image`:
+  - `freebsd_source_materializations_file`
+  - `materialized_source_store_count`
+  - `materialized_source_stores`
+- added validation artifacts:
+  - `tests/system/phase16-git-materialized-source-operating-system.scm.in`
+  - `tests/system/run-phase16-source-driven-native-build.sh`
+- wrote:
+  - `docs/reports/phase16-source-driven-native-builds-freebsd.md`
+
+Validation:
+
+- `PASS phase16-source-driven-native-build`
+- `PASS phase16-source-materialization`
+- `PASS phase16-declarative-source-build`
+- validated a full native system build from declared Git source:
+  - `https://git.FreeBSD.org/src.git`
+  - ref: `stable/15`
+  - resolved commit during validation:
+    - `332708a606f6bf0841c1d4a74c0d067f5640fe89`
+- intentionally declared an unused transitional source-root:
+  - `/var/empty/fruix-unused-source-root`
+  and confirmed native build info instead used the materialized source root under `/frx/store/*-freebsd-source-*/tree`
+
+Current assessment:
+
+- Phase 16 is now fully complete
+- Fruix can:
+  - declare FreeBSD source inputs
+  - materialize them under `/frx/store`
+  - build native FreeBSD base artifacts from those materialized source snapshots
+- the next step is Phase 17:
+  - build and compare side-by-side source revisions, then boot from them
+
 ## 2026-04-03 — Phase 16.2 completed: Fruix now materializes FreeBSD source inputs under its control
 
 Completed work:

docs/PROG_SUMMARY.md

@@ -27,6 +27,7 @@ Completed milestones include:
   - `https://git.FreeBSD.org/src.git`
   - official `src.txz` archives such as `https://download.freebsd.org/releases/amd64/15.0-RELEASE/src.txz`
   into `/frx/store`, with cache-backed provenance under `/frx/var/cache/fruix/freebsd-source`.
+- **Source-driven native base builds**: native FreeBSD kernel/bootloader/runtime artifacts now consume those materialized source snapshots rather than ambient `/usr/src`, and their build metadata records both the declared source and the effective materialized source identity.
 - **Base upgrade story**: Fruix can now keep distinct declared base versions side by side in `/frx/store` and roll forward / back between them through the normal system deployment flow.
 
 ## Major pain points now behind us
@@ -41,7 +42,7 @@ Completed milestones include:
 ## Major pain points still ahead
 
 - **True store-native runtime artifacts**: some historical build/install prefixes are still embedded in binaries and metadata. They are no longer required at runtime, but the local Guile/guile-extra/Shepherd build/install flow should still be moved to a genuinely store-native prefix from the start.
-- **Source-driven native base builds**: Fruix can now declare and materialize FreeBSD source inputs, but the validated native kernel/world/runtime build path still needs to consume those materialized source artifacts instead of ambient `/usr/src`.
+- **Side-by-side source revision validation**: Fruix now has the source boundary needed for this work, but it still needs stronger side-by-side revision boot validation and upgrade/rollback exercises across distinct fetched source revisions.
 - **Boot-path simplification**: Fruix now supports both the legacy `freebsd-init+rc.d-shepherd` path and the more Guix-like `shepherd-pid1` path. We still need to decide whether Shepherd PID 1 becomes the preferred/default architecture.
 - **Reduce transitional FreeBSD glue**: more of the current bootstrap/activation/runtime setup should become cleaner and less prototype-specific over time.
 - **Tooling and platform constraints**: local bhyve remains blocked by missing nested virtualization under Xen, and XO permissions still prevent creating/importing new VDIs; current validation must keep reusing the approved VM/VDI path.
@@ -49,4 +50,4 @@ Completed milestones include:
 
 ## Bottom line
 
-Fruix has crossed the most important threshold: it is no longer just a collection of isolated FreeBSD experiments. It can now build declarative FreeBSD system artifacts, boot them on the real target VM, reach the network, serve SSH, run Shepherd as PID 1, operate from `/frx` without depending on temporary runtime-prefix shims, build native FreeBSD base artifacts into `/frx/store`, roll forward / back between declared base versions, and materialize declared FreeBSD source inputs into `/frx/store`. The biggest remaining work is no longer “can this boot?” but “how fully can the native FreeBSD base path be driven by fetched, pinned, reproducible source inputs and a real installation/deployment story?”
+Fruix has crossed the most important threshold: it is no longer just a collection of isolated FreeBSD experiments. It can now build declarative FreeBSD system artifacts, boot them on the real target VM, reach the network, serve SSH, run Shepherd as PID 1, operate from `/frx` without depending on temporary runtime-prefix shims, build native FreeBSD base artifacts into `/frx/store`, roll forward / back between declared base versions, materialize declared FreeBSD source inputs into `/frx/store`, and drive native base builds from those materialized source snapshots. The biggest remaining work is no longer “can this boot?” but “how fully can those fetched, pinned source revisions be validated side by side and turned into a real installation/deployment story?”

docs/reports/phase16-source-driven-native-builds-freebsd.md

@@ -0,0 +1,182 @@
+# Phase 16.3: build native FreeBSD base artifacts from materialized source inputs
+
+Date: 2026-04-03
+
+## Goal
+
+Phase 16.3 closes the immediate gap between:
+
+- declarative/materialized FreeBSD source inputs, and
+- the native kernel/world/runtime/bootloader build path.
+
+Before this step, Fruix could already:
+
+- describe FreeBSD sources explicitly, and
+- materialize them into `/frx/store`
+
+but native base builds still consumed the ambient host source tree through the package plan's `source-root`.
+
+After this step, native FreeBSD base artifacts are driven by the declared source input's **materialized store snapshot** instead.
+
+## Implementation
+
+### Native package materialization now injects materialized source inputs
+
+`modules/fruix/system/freebsd.scm` now prepares native FreeBSD packages differently from copy-based packages.
+
+For native packages, Fruix now:
+
+1. reconstructs the declared `freebsd-source` from the package plan
+2. materializes that source under Fruix control
+3. rewrites the native build plan to use the materialized source root
+4. records both the declared source and the materialized/effective source in the output metadata
+5. adds the materialized source store path to the package references
+
+This means native package output identity now depends on:
+
+- the declared source object
+- the resolved/materialized source identity
+- the materialized source root and tree hash
+
+rather than on an ambient host tree path.
+
+### Native build manifests now record both declared and materialized source identity
+
+Native package manifests now include:
+
+- `declared-source`
+- `materialized-source`
+- the `native-build-common` block derived from the materialized source root
+
+Native `.freebsd-native-build-info.scm` files now record:
+
+- declared source
+- materialized source store path
+- materialized source root
+- materialized source tree hash
+- effective source details such as resolved Git commit or verified archive SHA256
+
+### Source tree hashing now reuses materialized-source metadata when available
+
+The native build common manifest now prefers the materialized source tree hash when one is already known, rather than recomputing it unnecessarily.
+
+### Package caching is now manifest-driven rather than name/version-only
+
+The in-process package materialization cache for native builds now keys on the computed manifest identity rather than only:
+
+- package name
+- package version
+
+That is important once multiple source-driven variants of the same native package version can coexist.
+
+### System closures now record source materializations explicitly
+
+`materialize-operating-system` now keeps a dedicated source-materialization cache while building native packages.
+
+Closures now record:
+
+- `metadata/freebsd-source-materializations.scm`
+- `materialized-source-store-count`
+- `materialized-source-stores`
+
+and system references now include the materialized source stores explicitly.
+
+This makes the source artifacts themselves part of the closure provenance boundary.
+
+### `fruix system` metadata now exposes materialized source stores
+
+`scripts/fruix.scm` now emits for `fruix system build` and `image`:
+
+- `freebsd_source_materializations_file`
+- `materialized_source_store_count`
+- `materialized_source_stores`
+
+This makes it easy for harnesses and operators to see which source store snapshot was actually used for a build.
+
+## New files
+
+Added:
+
+- `tests/system/phase16-git-materialized-source-operating-system.scm.in`
+- `tests/system/run-phase16-source-driven-native-build.sh`
+- `docs/reports/phase16-source-driven-native-builds-freebsd.md`
+
+## Validation
+
+Passing runs:
+
+- `PASS phase16-source-driven-native-build`
+- `PASS phase16-source-materialization`
+- `PASS phase16-declarative-source-build`
+
+### Source-driven native build validation
+
+Validated a native system build from a declared Git source:
+
+- source kind:
+  - `git`
+- source URL:
+  - `https://git.FreeBSD.org/src.git`
+- source ref:
+  - `stable/15`
+- resolved commit during validation:
+  - `332708a606f6bf0841c1d4a74c0d067f5640fe89`
+
+The test intentionally declared an **unused transitional source-root**:
+
+- `/var/empty/fruix-unused-source-root`
+
+and still built successfully, proving the native build path now uses the materialized source snapshot rather than the ambient declared root.
+
+Confirmed build metadata:
+
+```text
+closure_path=/frx/store/efed0b8ee31b4f3f97b8ae04317ccd2f3f8a88e332172e2bf0bb4295271c6023-fruix-system-fruix-freebsd
+kernel_store=/frx/store/c7e094a450f017a35c2d4d66451dcda893bc4ac27d810295fa491e41fdae8551-freebsd-native-kernel-15.0-STABLE-git-materialized
+bootloader_store=/frx/store/36432acf23cdff7eb7601bd314ea82d75a5e7fbcd3480d753117f4c5f9a7c7db-freebsd-native-bootloader-15.0-STABLE-git-materialized
+runtime_store=/frx/store/b6f9ebb5c844aedc3d4395ad08542eb67965a19fc28f68fb0f4b55f91aef1c8f-freebsd-native-runtime-15.0-STABLE-git-materialized
+materialized_source_store=/frx/store/bcbf7554ce43d40b1bdc98823eb450ce050529b34bbe2dae2a9df48688004c4b-freebsd-source-stable15-network-source
+materialized_source_root=/frx/store/bcbf7554ce43d40b1bdc98823eb450ce050529b34bbe2dae2a9df48688004c4b-freebsd-source-stable15-network-source/tree
+resolved_commit=332708a606f6bf0841c1d4a74c0d067f5640fe89
+source_driven_native_build=ok
+```
+
+The harness also verified that:
+
+- closure metadata records exactly one materialized source store
+- `metadata/freebsd-source-materializations.scm` exists
+- `metadata/store-layout.scm` records the materialized source store count/path
+- native build info files for kernel, bootloader, and runtime all record:
+  - a `declared-source` block with `ref . "stable/15"`
+  - a `materialized-source` block
+  - the materialized source store path
+  - the materialized source root
+  - the resolved Git commit
+- native build info files do **not** use the unused declared source root as their actual `source-root`
+
+### Regression checks
+
+Re-ran:
+
+- `PASS phase16-source-materialization`
+- `PASS phase16-declarative-source-build`
+
+The local-tree declarative source build now also drives native base outputs from a materialized local source snapshot in `/frx/store`, while preserving the same declared source metadata interface.
+
+## Result
+
+Phase 16.3 is complete.
+
+Fruix native FreeBSD base artifacts are now built from the declared source input's materialized store snapshot rather than directly from ambient `/usr/src`.
+
+That means Fruix has crossed the key source boundary planned for Phase 16:
+
+- source inputs can be declared
+- source inputs can be fetched/materialized
+- native base artifacts can now consume those materialized source inputs
+
+The next logical phase is no longer source modeling itself, but what to do with that stronger source boundary:
+
+- side-by-side source revisions
+- boot validation from distinct source revisions
+- and then installation/deployment ergonomics on top of those source-driven native builds

modules/fruix/packages/freebsd.scm

@@ -26,6 +26,7 @@
             freebsd-base-kernconf
             freebsd-base-make-flags
             %default-freebsd-base
+            freebsd-package
             freebsd-package?
             freebsd-package-name
             freebsd-package-version

modules/fruix/system/freebsd.scm

@@ -300,7 +300,8 @@
     `((build-version . ,native-freebsd-build-version)
       (source-root . ,source-root)
       (source-tree-identity-mode . "mtree:type,link,size,mode,sha256digest")
-      (source-tree-sha256 . ,(native-build-source-tree-sha256 source-root))
+      (source-tree-sha256 . ,(or (build-plan-ref plan 'materialized-source-tree-sha256 #f)
+                                 (native-build-source-tree-sha256 source-root)))
       (target . ,target)
       (target-arch . ,target-arch)
       (kernconf . ,kernconf)
@@ -323,11 +324,25 @@
     (commit . ,(build-plan-ref plan 'base-source-commit #f))
     (sha256 . ,(build-plan-ref plan 'base-source-sha256 #f))))
 
+(define (native-build-materialized-source plan)
+  `((store-path . ,(build-plan-ref plan 'materialized-source-store #f))
+    (source-root . ,(build-plan-ref plan 'source-root "/usr/src"))
+    (info-file . ,(build-plan-ref plan 'materialized-source-info-file #f))
+    (tree-sha256 . ,(build-plan-ref plan 'materialized-source-tree-sha256 #f))
+    (cache-path . ,(build-plan-ref plan 'materialized-source-cache-path #f))
+    (effective-source . ((kind . ,(build-plan-ref plan 'effective-source-kind #f))
+                         (url . ,(build-plan-ref plan 'effective-source-url #f))
+                         (path . ,(build-plan-ref plan 'effective-source-path #f))
+                         (ref . ,(build-plan-ref plan 'effective-source-ref #f))
+                         (commit . ,(build-plan-ref plan 'effective-source-commit #f))
+                         (sha256 . ,(build-plan-ref plan 'effective-source-sha256 #f))))))
+
 (define (native-build-manifest-string package input-paths)
   (let* ((plan (freebsd-package-install-plan package))
          (common (native-build-common-manifest plan))
          (declared-base (native-build-declared-base plan))
          (declared-source (native-build-declared-source plan))
+         (materialized-source (native-build-materialized-source plan))
          (keep-paths (build-plan-ref plan 'keep-paths '()))
          (prune-paths (build-plan-ref plan 'prune-paths '())))
     (string-append
@@ -339,6 +354,8 @@
      (object->string declared-base)
      "\ndeclared-source=\n"
      (object->string declared-source)
+     "\nmaterialized-source=\n"
+     (object->string materialized-source)
      "\nnative-build-common=\n"
      (object->string common)
      "\nkeep-paths=\n"
@@ -493,6 +510,7 @@
       (version . ,(freebsd-package-version package))
       (declared-base . ,(native-build-declared-base plan))
       (declared-source . ,(native-build-declared-source plan))
+      (materialized-source . ,(native-build-materialized-source plan))
       (build-system . ,(freebsd-package-build-system package))
       (source-root . ,(assoc-ref common 'source-root))
       (source-tree-sha256 . ,(assoc-ref common 'source-tree-sha256))
@@ -550,38 +568,101 @@
     (write-file (string-append output-path "/.freebsd-native-build-info.scm")
                 (object->string (native-build-output-metadata package common build-root final-stage-root)))))
 
-(define (package-cache-key package)
+(define (package-with-install-plan package install-plan)
-  (string-append (freebsd-package-name package) "-" (freebsd-package-version package)))
+  (freebsd-package
+   #:name (freebsd-package-name package)
+   #:version (freebsd-package-version package)
+   #:build-system (freebsd-package-build-system package)
+   #:inputs (freebsd-package-inputs package)
+   #:home-page (freebsd-package-home-page package)
+   #:synopsis (freebsd-package-synopsis package)
+   #:description (freebsd-package-description package)
+   #:license (freebsd-package-license package)
+   #:install-plan install-plan))
 
-(define (materialize-freebsd-package package store-dir cache)
+(define (plan-freebsd-source plan)
-  (let ((cached (hash-ref cache (package-cache-key package) #f)))
+  (freebsd-source #:name (build-plan-ref plan 'base-source-name "default")
+                  #:kind (build-plan-ref plan 'base-source-kind 'local-tree)
+                  #:url (build-plan-ref plan 'base-source-url #f)
+                  #:path (build-plan-ref plan 'base-source-path #f)
+                  #:ref (build-plan-ref plan 'base-source-ref #f)
+                  #:commit (build-plan-ref plan 'base-source-commit #f)
+                  #:sha256 (build-plan-ref plan 'base-source-sha256 #f)))
+
+(define (source-cache-key source)
+  (string-hash (object->string (freebsd-source-spec source))))
+
+(define (materialize-freebsd-source/cached source store-dir source-cache)
+  (let* ((key (source-cache-key source))
+         (cached (hash-ref source-cache key #f)))
+    (or cached
+        (let ((result (materialize-freebsd-source source #:store-dir store-dir)))
+          (hash-set! source-cache key result)
+          result))))
+
+(define (plan-with-materialized-source plan source-result)
+  (let* ((effective (assoc-ref source-result 'effective-source))
+         (overrides
+          `((source-root . ,(assoc-ref source-result 'source-root))
+            (materialized-source-store . ,(assoc-ref source-result 'source-store-path))
+            (materialized-source-info-file . ,(assoc-ref source-result 'source-info-file))
+            (materialized-source-tree-sha256 . ,(assoc-ref source-result 'source-tree-sha256))
+            (materialized-source-cache-path . ,(assoc-ref source-result 'cache-path))
+            (effective-source-kind . ,(assoc-ref effective 'kind))
+            (effective-source-url . ,(assoc-ref effective 'url))
+            (effective-source-path . ,(assoc-ref effective 'path))
+            (effective-source-ref . ,(assoc-ref effective 'ref))
+            (effective-source-commit . ,(assoc-ref effective 'commit))
+            (effective-source-sha256 . ,(assoc-ref effective 'sha256)))))
+    (append overrides plan)))
+
+(define* (materialize-freebsd-package package store-dir cache #:optional source-cache)
+  (let* ((source-cache (or source-cache (make-hash-table)))
+         (input-paths (map (lambda (input)
+                             (materialize-freebsd-package input store-dir cache source-cache))
+                           (freebsd-package-inputs package)))
+         (prepared-package
+          (if (freebsd-native-build-package? package)
+              (let* ((source (plan-freebsd-source (freebsd-package-install-plan package)))
+                     (source-result (materialize-freebsd-source/cached source store-dir source-cache))
+                     (plan (plan-with-materialized-source (freebsd-package-install-plan package)
+                                                         source-result)))
+                (package-with-install-plan package plan))
+              package))
+         (effective-input-paths
+          (if (freebsd-native-build-package? package)
+              (cons (build-plan-ref (freebsd-package-install-plan prepared-package)
+                                    'materialized-source-store
+                                    #f)
+                    input-paths)
+              input-paths))
+         (effective-input-paths (filter identity effective-input-paths))
+         (manifest (package-manifest-string prepared-package effective-input-paths))
+         (cache-key (string-hash manifest))
+         (cached (hash-ref cache cache-key #f)))
     (if cached
         cached
-        (let* ((input-paths (map (lambda (input)
+        (let* ((hash (string-hash manifest))
-                                   (materialize-freebsd-package input store-dir cache))
-                                 (freebsd-package-inputs package)))
-               (manifest (package-manifest-string package input-paths))
-               (hash (string-hash manifest))
                (output-path (string-append store-dir "/" hash "-"
-                                           (freebsd-package-name package)
+                                           (freebsd-package-name prepared-package)
                                            "-"
-                                           (freebsd-package-version package))))
+                                           (freebsd-package-version prepared-package))))
           (unless (file-exists? output-path)
-            (case (freebsd-package-build-system package)
+            (case (freebsd-package-build-system prepared-package)
               ((copy-build-system)
                (mkdir-p output-path)
                (for-each (lambda (entry)
                            (materialize-plan-entry output-path entry))
-                         (freebsd-package-install-plan package))
+                         (freebsd-package-install-plan prepared-package))
                (write-file (string-append output-path "/.references")
-                           (string-join input-paths "\n"))
+                           (string-join effective-input-paths "\n"))
                (write-file (string-append output-path "/.fruix-package") manifest))
               ((freebsd-world-build-system freebsd-kernel-build-system)
-               (materialize-native-freebsd-package package input-paths manifest output-path))
+               (materialize-native-freebsd-package prepared-package effective-input-paths manifest output-path))
               (else
                (error (format #f "unsupported package build system: ~a"
-                              (freebsd-package-build-system package))))))
+                              (freebsd-package-build-system prepared-package))))))
-          (hash-set! cache (package-cache-key package) output-path)
+          (hash-set! cache cache-key output-path)
           output-path))))
 
 (define prefix-materializer-version "3")
@@ -1572,6 +1653,20 @@
   (mkdir-p tree-root)
   (walk ""))
 
+(define (hash-table-values table)
+  (hash-fold (lambda (_ value result)
+               (cons value result))
+             '()
+             table))
+
+(define (freebsd-source-materialization-spec result)
+  `((source-store-path . ,(assoc-ref result 'source-store-path))
+    (source-root . ,(assoc-ref result 'source-root))
+    (source-info-file . ,(assoc-ref result 'source-info-file))
+    (source-tree-sha256 . ,(assoc-ref result 'source-tree-sha256))
+    (cache-path . ,(assoc-ref result 'cache-path))
+    (effective-source . ,(assoc-ref result 'effective-source))))
+
 (define* (materialize-operating-system os
                                        #:key
                                        (store-dir "/frx/store")
@@ -1580,13 +1675,14 @@
                                        (shepherd-prefix "/tmp/shepherd-freebsd-validate-install"))
   (validate-operating-system os)
   (let* ((cache (make-hash-table))
+         (source-cache (make-hash-table))
          (kernel-package (operating-system-kernel os))
          (bootloader-package (operating-system-bootloader os))
          (base-packages (operating-system-base-packages os))
-         (kernel-store (materialize-freebsd-package kernel-package store-dir cache))
+         (kernel-store (materialize-freebsd-package kernel-package store-dir cache source-cache))
-         (bootloader-store (materialize-freebsd-package bootloader-package store-dir cache))
+         (bootloader-store (materialize-freebsd-package bootloader-package store-dir cache source-cache))
          (base-package-stores (map (lambda (package)
-                                     (materialize-freebsd-package package store-dir cache))
+                                     (materialize-freebsd-package package store-dir cache source-cache))
                                    base-packages))
          (base-package-pairs (map cons base-packages base-package-stores))
          (store-classification
@@ -1627,17 +1723,27 @@
                           (freebsd-native-build-package? (car entry)))
                         store-classification))))
          (fruix-runtime-stores (list guile-store guile-extra-store shepherd-store))
+         (source-materializations
+          (delete-duplicates (hash-table-values source-cache)))
+         (materialized-source-stores
+          (delete-duplicates (map (lambda (result)
+                                    (assoc-ref result 'source-store-path))
+                                  source-materializations)))
          (metadata-files
           `(("metadata/freebsd-base.scm"
              . ,(object->string (freebsd-base-spec (operating-system-freebsd-base os))))
             ("metadata/freebsd-source.scm"
              . ,(object->string (freebsd-source-spec (freebsd-base-source (operating-system-freebsd-base os)))))
+            ("metadata/freebsd-source-materializations.scm"
+             . ,(object->string (map freebsd-source-materialization-spec source-materializations)))
             ("metadata/host-base-provenance.scm"
              . ,(object->string (host-freebsd-provenance)))
             ("metadata/store-layout.scm"
              . ,(object->string
                  `((freebsd-base . ,(freebsd-base-spec (operating-system-freebsd-base os)))
                    (freebsd-source . ,(freebsd-source-spec (freebsd-base-source (operating-system-freebsd-base os))))
+                   (materialized-source-store-count . ,(length materialized-source-stores))
+                   (materialized-source-stores . ,materialized-source-stores)
                    (host-base-store-count . ,(length host-base-stores))
                    (host-base-stores . ,host-base-stores)
                    (native-base-store-count . ,(length native-base-stores))
@@ -1655,7 +1761,7 @@
                                      . ,(render-activation-rc-script))
                                     ("usr/local/etc/rc.d/fruix-shepherd"
                                      . ,(render-rc-script shepherd-store guile-store guile-extra-store)))))
-         (references (delete-duplicates (append host-base-stores native-base-stores fruix-runtime-stores)))
+         (references (delete-duplicates (append materialized-source-stores host-base-stores native-base-stores fruix-runtime-stores)))
          (manifest (string-append
                     "closure-spec=\n"
                     (object->string (operating-system-closure-spec os))
@@ -1714,6 +1820,8 @@
       (fruix-runtime-stores . ,fruix-runtime-stores)
       (freebsd-base-file . ,(string-append closure-path "/metadata/freebsd-base.scm"))
       (freebsd-source-file . ,(string-append closure-path "/metadata/freebsd-source.scm"))
+      (freebsd-source-materializations-file . ,(string-append closure-path "/metadata/freebsd-source-materializations.scm"))
+      (materialized-source-stores . ,materialized-source-stores)
       (host-base-provenance-file . ,(string-append closure-path "/metadata/host-base-provenance.scm"))
       (store-layout-file . ,(string-append closure-path "/metadata/store-layout.scm"))
       (generated-files . ,(map car generated-files))
@@ -1981,6 +2089,9 @@
       (native-base-stores . ,(assoc-ref closure 'native-base-stores))
       (fruix-runtime-stores . ,(assoc-ref closure 'fruix-runtime-stores))
       (freebsd-base-file . ,(assoc-ref closure 'freebsd-base-file))
+      (freebsd-source-file . ,(assoc-ref closure 'freebsd-source-file))
+      (freebsd-source-materializations-file . ,(assoc-ref closure 'freebsd-source-materializations-file))
+      (materialized-source-stores . ,(assoc-ref closure 'materialized-source-stores))
       (host-base-provenance-file . ,(assoc-ref closure 'host-base-provenance-file))
       (store-layout-file . ,(assoc-ref closure 'store-layout-file))
       (image-spec . ,image-spec)

scripts/fruix.scm

@@ -249,6 +249,9 @@ Common options:\n\
        (freebsd_source_commit . ,(or (freebsd-source-commit source) ""))
        (freebsd_source_sha256 . ,(or (freebsd-source-sha256 source) ""))
        (freebsd_source_file . ,(assoc-ref result 'freebsd-source-file))
+       (freebsd_source_materializations_file . ,(assoc-ref result 'freebsd-source-materializations-file))
+       (materialized_source_store_count . ,(length (assoc-ref result 'materialized-source-stores)))
+       (materialized_source_stores . ,(string-join (assoc-ref result 'materialized-source-stores) ","))
        (ready_marker . ,(operating-system-ready-marker os))
        (kernel_store . ,(assoc-ref result 'kernel-store))
        (bootloader_store . ,(assoc-ref result 'bootloader-store))
@@ -304,6 +307,9 @@ Common options:\n\
        (freebsd_source_commit . ,(or (freebsd-source-commit source) ""))
        (freebsd_source_sha256 . ,(or (freebsd-source-sha256 source) ""))
        (freebsd_source_file . ,(assoc-ref result 'freebsd-source-file))
+       (freebsd_source_materializations_file . ,(assoc-ref result 'freebsd-source-materializations-file))
+       (materialized_source_store_count . ,(length (assoc-ref result 'materialized-source-stores)))
+       (materialized_source_stores . ,(string-join (assoc-ref result 'materialized-source-stores) ","))
        (disk_capacity . ,(assoc-ref image-spec 'disk-capacity))
        (root_size . ,(assoc-ref image-spec 'root-size))
        (image_store_path . ,(assoc-ref result 'image-store-path))

tests/system/phase16-git-materialized-source-operating-system.scm.in

@@ -0,0 +1,90 @@
+(use-modules (fruix system freebsd)
+             (fruix packages freebsd))
+
+(define phase16-source
+  (freebsd-source
+   #:name "__SOURCE_NAME__"
+   #:kind 'git
+   #:ref "__SOURCE_REF__"))
+
+(define phase16-base
+  (freebsd-base
+   #:name "__BASE_NAME__"
+   #:version-label "__BASE_VERSION_LABEL__"
+   #:release "__BASE_RELEASE__"
+   #:branch "__BASE_BRANCH__"
+   #:source phase16-source
+   #:source-root "__DECLARED_SOURCE_ROOT__"
+   #:target "amd64"
+   #:target-arch "amd64"
+   #:kernconf "GENERIC"))
+
+(define phase16-operating-system
+  (operating-system
+   #:host-name "fruix-freebsd"
+   #:freebsd-base phase16-base
+   #:kernel (freebsd-native-kernel-for phase16-base)
+   #:bootloader (freebsd-native-bootloader-for phase16-base)
+   #:base-packages (freebsd-native-system-packages-for phase16-base)
+   #:groups (list (user-group #:name "wheel" #:gid 0 #:system? #t)
+                  (user-group #:name "sshd" #:gid 22 #:system? #t)
+                  (user-group #:name "_dhcp" #:gid 65 #:system? #t)
+                  (user-group #:name "operator" #:gid 1000 #:system? #f))
+   #:users (list (user-account #:name "root"
+                               #:uid 0
+                               #:group "wheel"
+                               #:comment "Charlie &"
+                               #:home "/root"
+                               #:shell "/bin/sh"
+                               #:system? #t)
+                 (user-account #:name "sshd"
+                               #:uid 22
+                               #:group "sshd"
+                               #:comment "Secure Shell Daemon"
+                               #:home "/var/empty"
+                               #:shell "/usr/sbin/nologin"
+                               #:system? #t)
+                 (user-account #:name "_dhcp"
+                               #:uid 65
+                               #:group "_dhcp"
+                               #:comment "dhcp programs"
+                               #:home "/var/empty"
+                               #:shell "/usr/sbin/nologin"
+                               #:system? #t)
+                 (user-account #:name "operator"
+                               #:uid 1000
+                               #:group "operator"
+                               #:supplementary-groups '("wheel")
+                               #:comment "Fruix Operator"
+                               #:home "/home/operator"
+                               #:shell "/bin/sh"
+                               #:system? #f))
+   #:file-systems (list (file-system #:device "/dev/gpt/fruix-root"
+                                     #:mount-point "/"
+                                     #:type "ufs"
+                                     #:options "rw"
+                                     #:needed-for-boot? #t)
+                        (file-system #:device "devfs"
+                                     #:mount-point "/dev"
+                                     #:type "devfs"
+                                     #:options "rw"
+                                     #:needed-for-boot? #t)
+                        (file-system #:device "tmpfs"
+                                     #:mount-point "/tmp"
+                                     #:type "tmpfs"
+                                     #:options "rw,size=64m"))
+   #:services '(shepherd ready-marker sshd)
+   #:loader-entries '(("autoboot_delay" . "1")
+                      ("boot_multicons" . "YES")
+                      ("boot_serial" . "YES")
+                      ("console" . "comconsole,vidconsole"))
+   #:rc-conf-entries '(("clear_tmp_enable" . "NO")
+                       ("hostid_enable" . "NO")
+                       ("sendmail_enable" . "NONE")
+                       ("sshd_enable" . "YES")
+                       ("ifconfig_xn0" . "SYNCDHCP")
+                       ("ifconfig_em0" . "SYNCDHCP")
+                       ("ifconfig_vtnet0" . "SYNCDHCP"))
+   #:init-mode 'shepherd-pid1
+   #:ready-marker "/var/lib/fruix/ready"
+   #:root-authorized-keys '("__ROOT_AUTHORIZED_KEY__")))

tests/system/run-phase13-native-base-build.sh

@@ -139,13 +139,29 @@ grep -F 'native-base-stores' "$store_layout_file" >/dev/null || {
   exit 1
 }
 
-grep -F 'source-root . "/usr/src"' "$world_build_info" >/dev/null || {
+grep -F '(materialized-source' "$world_build_info" >/dev/null || {
-  echo "world build info is missing /usr/src provenance" >&2
+  echo "world build info is missing materialized source metadata" >&2
+  exit 1
+}
+grep -F 'path . "/usr/src"' "$world_build_info" >/dev/null || {
+  echo "world build info is missing declared local-tree source provenance" >&2
+  exit 1
+}
+grep -F 'source-root . "/frx/store/' "$world_build_info" >/dev/null || {
+  echo "world build info is not using a materialized source root" >&2
   exit 1
 }
 
-grep -F 'source-root . "/usr/src"' "$kernel_build_info" >/dev/null || {
+grep -F '(materialized-source' "$kernel_build_info" >/dev/null || {
-  echo "kernel build info is missing /usr/src provenance" >&2
+  echo "kernel build info is missing materialized source metadata" >&2
+  exit 1
+}
+grep -F 'path . "/usr/src"' "$kernel_build_info" >/dev/null || {
+  echo "kernel build info is missing declared local-tree source provenance" >&2
+  exit 1
+}
+grep -F 'source-root . "/frx/store/' "$kernel_build_info" >/dev/null || {
+  echo "kernel build info is not using a materialized source root" >&2
   exit 1
 }
 

tests/system/run-phase16-source-driven-native-build.sh

@@ -0,0 +1,252 @@
+#!/bin/sh
+set -eu
+
+project_root=${PROJECT_ROOT:-$(pwd)}
+script_dir=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
+fruix_cmd=$project_root/bin/fruix
+os_template=${OS_TEMPLATE:-$script_dir/phase16-git-materialized-source-operating-system.scm.in}
+system_name=${SYSTEM_NAME:-phase16-operating-system}
+store_dir=${STORE_DIR:-/frx/store}
+base_name=${BASE_NAME:-git-materialized-source}
+base_version_label=${BASE_VERSION_LABEL:-15.0-STABLE-git-materialized}
+base_release=${BASE_RELEASE:-15.0-STABLE}
+base_branch=${BASE_BRANCH:-stable/15}
+source_name=${SOURCE_NAME:-stable15-network-source}
+source_ref=${SOURCE_REF:-stable/15}
+declared_source_root=${DECLARED_SOURCE_ROOT:-/var/empty/fruix-unused-source-root}
+metadata_target=${METADATA_OUT:-}
+root_authorized_key_file=${ROOT_AUTHORIZED_KEY_FILE:-$HOME/.ssh/id_ed25519.pub}
+
+[ -x "$fruix_cmd" ] || {
+  echo "fruix command is not executable: $fruix_cmd" >&2
+  exit 1
+}
+[ -f "$os_template" ] || {
+  echo "missing operating-system template: $os_template" >&2
+  exit 1
+}
+[ -f "$root_authorized_key_file" ] || {
+  echo "missing root authorized key file: $root_authorized_key_file" >&2
+  exit 1
+}
+
+cleanup=0
+if [ -n "${WORKDIR:-}" ]; then
+  workdir=$WORKDIR
+  mkdir -p "$workdir"
+else
+  workdir=$(mktemp -d /tmp/fruix-phase16-source-native-build.XXXXXX)
+  cleanup=1
+fi
+if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then
+  cleanup=0
+fi
+
+cleanup_workdir() {
+  if [ "$cleanup" -eq 1 ]; then
+    rm -rf "$workdir" 2>/dev/null || sudo rm -rf "$workdir"
+  fi
+}
+trap cleanup_workdir EXIT INT TERM
+
+phase16_os_file=$workdir/phase16-git-materialized-source-operating-system.scm
+build_out=$workdir/build.txt
+metadata_file=$workdir/phase16-source-driven-native-build-metadata.txt
+root_authorized_key=$(tr -d '\n' < "$root_authorized_key_file")
+sed \
+  -e "s|__BASE_NAME__|$base_name|g" \
+  -e "s|__BASE_VERSION_LABEL__|$base_version_label|g" \
+  -e "s|__BASE_RELEASE__|$base_release|g" \
+  -e "s|__BASE_BRANCH__|$base_branch|g" \
+  -e "s|__SOURCE_NAME__|$source_name|g" \
+  -e "s|__SOURCE_REF__|$source_ref|g" \
+  -e "s|__DECLARED_SOURCE_ROOT__|$declared_source_root|g" \
+  -e "s|__ROOT_AUTHORIZED_KEY__|$root_authorized_key|g" \
+  "$os_template" > "$phase16_os_file"
+
+action_env() {
+  sudo env \
+    HOME="$HOME" \
+    GUILE_AUTO_COMPILE=0 \
+    FRUIX_FREEBSD_BUILD_JOBS="${FRUIX_FREEBSD_BUILD_JOBS:-8}" \
+    GUIX_SOURCE_DIR="${GUIX_SOURCE_DIR:-$HOME/repos/guix}" \
+    GUILE_BIN="${GUILE_BIN:-/tmp/guile-freebsd-validate-install/bin/guile}" \
+    GUILE_EXTRA_PREFIX="${GUILE_EXTRA_PREFIX:-/tmp/guile-gnutls-freebsd-validate-install}" \
+    SHEPHERD_PREFIX="${SHEPHERD_PREFIX:-/tmp/shepherd-freebsd-validate-install}" \
+    "$@"
+}
+
+action_env "$fruix_cmd" system build "$phase16_os_file" --system "$system_name" --store "$store_dir" >"$build_out"
+
+field() {
+  sed -n "s/^$1=//p" "$build_out" | tail -n 1
+}
+
+closure_path=$(field closure_path)
+kernel_store=$(field kernel_store)
+bootloader_store=$(field bootloader_store)
+native_base_store_count=$(field native_base_store_count)
+native_base_stores=$(field native_base_stores)
+host_base_store_count=$(field host_base_store_count)
+freebsd_base_name_out=$(field freebsd_base_name)
+freebsd_base_version_label_out=$(field freebsd_base_version_label)
+freebsd_base_release_out=$(field freebsd_base_release)
+freebsd_base_branch_out=$(field freebsd_base_branch)
+freebsd_base_source_root_out=$(field freebsd_base_source_root)
+freebsd_base_file=$(field freebsd_base_file)
+freebsd_source_name_out=$(field freebsd_source_name)
+freebsd_source_kind_out=$(field freebsd_source_kind)
+freebsd_source_url_out=$(field freebsd_source_url)
+freebsd_source_ref_out=$(field freebsd_source_ref)
+freebsd_source_file=$(field freebsd_source_file)
+freebsd_source_materializations_file=$(field freebsd_source_materializations_file)
+materialized_source_store_count=$(field materialized_source_store_count)
+materialized_source_stores=$(field materialized_source_stores)
+store_layout_file=$(field store_layout_file)
+
+[ -n "$closure_path" ] || { echo "missing closure path" >&2; exit 1; }
+[ "$host_base_store_count" = 0 ] || { echo "expected zero host base stores, got: $host_base_store_count" >&2; exit 1; }
+[ "$native_base_store_count" = 3 ] || { echo "expected three native base stores, got: $native_base_store_count" >&2; exit 1; }
+[ "$materialized_source_store_count" = 1 ] || { echo "expected one materialized source store, got: $materialized_source_store_count" >&2; exit 1; }
+[ "$freebsd_base_name_out" = "$base_name" ] || { echo "unexpected freebsd base name: $freebsd_base_name_out" >&2; exit 1; }
+[ "$freebsd_base_version_label_out" = "$base_version_label" ] || { echo "unexpected freebsd base version label: $freebsd_base_version_label_out" >&2; exit 1; }
+[ "$freebsd_base_release_out" = "$base_release" ] || { echo "unexpected freebsd base release: $freebsd_base_release_out" >&2; exit 1; }
+[ "$freebsd_base_branch_out" = "$base_branch" ] || { echo "unexpected freebsd base branch: $freebsd_base_branch_out" >&2; exit 1; }
+[ "$freebsd_base_source_root_out" = "$declared_source_root" ] || { echo "unexpected declared source root: $freebsd_base_source_root_out" >&2; exit 1; }
+[ "$freebsd_source_name_out" = "$source_name" ] || { echo "unexpected freebsd source name: $freebsd_source_name_out" >&2; exit 1; }
+[ "$freebsd_source_kind_out" = git ] || { echo "unexpected freebsd source kind: $freebsd_source_kind_out" >&2; exit 1; }
+[ "$freebsd_source_url_out" = https://git.FreeBSD.org/src.git ] || { echo "unexpected freebsd source URL: $freebsd_source_url_out" >&2; exit 1; }
+[ "$freebsd_source_ref_out" = "$source_ref" ] || { echo "unexpected freebsd source ref: $freebsd_source_ref_out" >&2; exit 1; }
+[ -f "$freebsd_base_file" ] || { echo "missing freebsd base file: $freebsd_base_file" >&2; exit 1; }
+[ -f "$freebsd_source_file" ] || { echo "missing freebsd source file: $freebsd_source_file" >&2; exit 1; }
+[ -f "$freebsd_source_materializations_file" ] || { echo "missing source materializations file: $freebsd_source_materializations_file" >&2; exit 1; }
+[ -f "$store_layout_file" ] || { echo "missing store layout file: $store_layout_file" >&2; exit 1; }
+
+case "$kernel_store" in
+  /frx/store/*-freebsd-native-kernel-$base_version_label) : ;;
+  *) echo "unexpected kernel store path: $kernel_store" >&2; exit 1 ;;
+esac
+case "$bootloader_store" in
+  /frx/store/*-freebsd-native-bootloader-$base_version_label) : ;;
+  *) echo "unexpected bootloader store path: $bootloader_store" >&2; exit 1 ;;
+esac
+case "$materialized_source_stores" in
+  /frx/store/*-freebsd-source-$source_name) : ;;
+  *) echo "unexpected materialized source store path: $materialized_source_stores" >&2; exit 1 ;;
+esac
+
+materialized_source_root=$materialized_source_stores/tree
+[ -f "$materialized_source_root/Makefile" ] || { echo "materialized git source root missing Makefile" >&2; exit 1; }
+[ -f "$materialized_source_root/sys/conf/newvers.sh" ] || { echo "materialized git source root missing newvers.sh" >&2; exit 1; }
+
+runtime_store=$(printf '%s\n' "$native_base_stores" | tr ',' '\n' | grep "freebsd-native-runtime-$base_version_label$" | head -n 1)
+[ -n "$runtime_store" ] || { echo "failed to recover runtime store" >&2; exit 1; }
+
+resolved_commit=$(grep -Eo '\(commit \. "[0-9a-f]{40}"\)' "$freebsd_source_materializations_file" | head -n 1 | sed 's/.*"\([0-9a-f]\{40\}\)".*/\1/')
+printf '%s\n' "$resolved_commit" | grep -E '^[0-9a-f]{40}$' >/dev/null || {
+  echo "failed to recover resolved git commit from source materializations file" >&2
+  exit 1
+}
+
+grep -F "$materialized_source_stores" "$freebsd_source_materializations_file" >/dev/null || {
+  echo "source materializations file missing store path" >&2
+  exit 1
+}
+grep -F "$materialized_source_root" "$freebsd_source_materializations_file" >/dev/null || {
+  echo "source materializations file missing source root" >&2
+  exit 1
+}
+grep -F "(materialized-source-store-count . 1)" "$store_layout_file" >/dev/null || {
+  echo "store layout file missing materialized source count" >&2
+  exit 1
+}
+grep -F "$materialized_source_stores" "$store_layout_file" >/dev/null || {
+  echo "store layout file missing materialized source store path" >&2
+  exit 1
+}
+
+for path in "$kernel_store/.freebsd-native-build-info.scm" "$bootloader_store/.freebsd-native-build-info.scm" "$runtime_store/.freebsd-native-build-info.scm"; do
+  [ -f "$path" ] || {
+    echo "missing native build info file: $path" >&2
+    exit 1
+  }
+  grep -F "(declared-source" "$path" >/dev/null || {
+    echo "native build info missing declared-source block in $path" >&2
+    exit 1
+  }
+  grep -F "(materialized-source" "$path" >/dev/null || {
+    echo "native build info missing materialized-source block in $path" >&2
+    exit 1
+  }
+  grep -F "(ref . \"$source_ref\")" "$path" >/dev/null || {
+    echo "native build info missing declared source ref in $path" >&2
+    exit 1
+  }
+  grep -F "(source-root . \"$materialized_source_root\")" "$path" >/dev/null || {
+    echo "native build info missing materialized source root in $path" >&2
+    exit 1
+  }
+  grep -F "(store-path . \"$materialized_source_stores\")" "$path" >/dev/null || {
+    echo "native build info missing materialized source store path in $path" >&2
+    exit 1
+  }
+  grep -F "(commit . \"$resolved_commit\")" "$path" >/dev/null || {
+    echo "native build info missing resolved git commit in $path" >&2
+    exit 1
+  }
+  if grep -F "(source-root . \"$declared_source_root\")" "$path" >/dev/null; then
+    echo "native build info still records the unused declared source root in $path" >&2
+    exit 1
+  fi
+done
+
+grep -F "(source-root . \"$declared_source_root\")" "$closure_path/parameters.scm" >/dev/null || {
+  echo "closure parameters no longer record the declared transitional source root" >&2
+  exit 1
+}
+
+closure_base=$(basename "$closure_path")
+cat >"$metadata_file" <<EOF
+workdir=$workdir
+phase16_os_file=$phase16_os_file
+closure_path=$closure_path
+closure_base=$closure_base
+kernel_store=$kernel_store
+bootloader_store=$bootloader_store
+runtime_store=$runtime_store
+native_base_store_count=$native_base_store_count
+native_base_stores=$native_base_stores
+host_base_store_count=$host_base_store_count
+freebsd_base_name=$freebsd_base_name_out
+freebsd_base_version_label=$freebsd_base_version_label_out
+freebsd_base_release=$freebsd_base_release_out
+freebsd_base_branch=$freebsd_base_branch_out
+freebsd_base_source_root=$freebsd_base_source_root_out
+freebsd_base_file=$freebsd_base_file
+freebsd_source_name=$freebsd_source_name_out
+freebsd_source_kind=$freebsd_source_kind_out
+freebsd_source_url=$freebsd_source_url_out
+freebsd_source_ref=$freebsd_source_ref_out
+freebsd_source_file=$freebsd_source_file
+freebsd_source_materializations_file=$freebsd_source_materializations_file
+materialized_source_store_count=$materialized_source_store_count
+materialized_source_store=$materialized_source_stores
+materialized_source_root=$materialized_source_root
+resolved_commit=$resolved_commit
+store_layout_file=$store_layout_file
+source_driven_native_build=ok
+EOF
+
+if [ -n "$metadata_target" ]; then
+  mkdir -p "$(dirname "$metadata_target")"
+  cp "$metadata_file" "$metadata_target"
+fi
+
+printf 'PASS phase16-source-driven-native-build\n'
+printf 'Work directory: %s\n' "$workdir"
+printf 'Metadata file: %s\n' "$metadata_file"
+if [ -n "$metadata_target" ]; then
+  printf 'Copied metadata to: %s\n' "$metadata_target"
+fi
+printf '%s\n' '--- metadata ---'
+cat "$metadata_file"