From 245a47dfbc8d32ac01232231f81511f3889e5e80 Mon Sep 17 00:00:00 2001 From: Steffen Beyer Date: Wed, 1 Apr 2026 09:58:56 +0200 Subject: [PATCH] Document gaps to real Guix FreeBSD builds --- docs/PROGRESS.md | 64 ++++ .../reports/phase1-guix-build-gap-analysis.md | 298 ++++++++++++++++++ 2 files changed, 362 insertions(+) create mode 100644 docs/reports/phase1-guix-build-gap-analysis.md diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 8f00734..88b0e6c 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -302,3 +302,67 @@ Next recommended step: 1. document the concrete remaining gap between these Scheme-driven phase-runner prototypes and a true Guix package/derivation/store-daemon build on FreeBSD, especially around store management, implicit inputs, and build isolation 2. or choose a somewhat more demanding GNU package with non-trivial declared inputs to identify the first builder-side FreeBSD adaptation points 3. continue keeping `~/repos/bdwgc` in reserve if later FreeBSD-specific GC/thread issues appear + +## 2026-04-01 — Phase 1.2 follow-up: documented the gap to a real Guix package build + +Completed work: + +- analyzed the concrete remaining gap between the current FreeBSD validation harnesses and a real Guix package/derivation/store-daemon build +- wrote the analysis to: + - `docs/reports/phase1-guix-build-gap-analysis.md` +- based the analysis on the current local Guix source tree, including the relevant host-side and daemon-side code paths in: + - `guix/packages.scm` + - `guix/build-system/gnu.scm` + - `guix/gexp.scm` + - `guix/store.scm` + - `guix/derivations.scm` + - `gnu/packages/commencement.scm` + - `doc/guix.texi` + - `nix/libstore/build.cc` +- compared those real Guix layers against the currently validated FreeBSD prototypes: + - shell-driven native GNU Hello harness + - Scheme-driven GNU Hello builder-phase runner + - Scheme-driven GNU which builder-phase runner + +Main conclusions recorded: + +- the current FreeBSD work has validated a narrow but real builder-side slice of Guix execution: + - Guile can run the needed Scheme code when using the fixed local build + - `(guix build gnu-build-system)` phases can build small GNU packages on FreeBSD +- however, the current prototypes still bypass several critical layers of a real Guix build: + - package -> bag lowering + - bag -> derivation lowering + - imported module closure/store materialization + - real daemon RPC and build submission + - canonical `/gnu/store` management and metadata + - build users, chroot/container or jail-style isolation + - substitute/graft/offload handling +- documented that the current phase runners rely on host tools already present on FreeBSD, whereas real `gnu-build-system` uses implicit inputs drawn from `%final-inputs` +- identified the most actionable next milestone as a derivation-generation investigation for a tiny package, to locate the first failure boundary among: + - host-side lowering + - store interaction + - derivation emission + - daemon availability + - daemon-side execution + +Current assessment: + +- Phase 1.2 now has both practical build validation and a clearer architectural map of what remains before a true Guix package build can work on FreeBSD +- the project has reduced uncertainty around builder-side GNU phase portability +- the next uncertainty is now specifically the host-side lowering/store/daemon boundary, not the builder-phase boundary + +Recent commits: + +- `e380e88` — `Add FreeBSD Guile verification harness` +- `cd721b1` — `Update progress after Guile verification` +- `27916cb` — `Diagnose Guile subprocess crash on FreeBSD` +- `02f7a7f` — `Validate local Guile fix on FreeBSD` +- `4aebea4` — `Add native GNU Hello FreeBSD build harness` +- `c944cdb` — `Validate Guix builder phases on FreeBSD` +- `0a2e48e` — `Validate GNU which builder phases on FreeBSD` + +Next recommended step: + +1. investigate whether a tiny package can be lowered far enough on FreeBSD to produce a real derivation, and capture the exact first failure point +2. if derivation generation proves immediately blocked, document whether the blocker is generated Guix modules/configuration, store connectivity, or daemon assumptions +3. continue keeping `~/repos/bdwgc` in reserve if later FreeBSD-specific GC/thread issues appear diff --git a/docs/reports/phase1-guix-build-gap-analysis.md b/docs/reports/phase1-guix-build-gap-analysis.md new file mode 100644 index 0000000..a07b220 --- /dev/null +++ b/docs/reports/phase1-guix-build-gap-analysis.md @@ -0,0 +1,298 @@ +# Phase 1.2 follow-up: gap analysis between current FreeBSD prototypes and a real Guix package build + +Date: 2026-04-01 + +## Summary + +This step documents the concrete remaining gap between the current FreeBSD validation harnesses and a true Guix package/derivation/store-daemon build. + +The repository now has three relevant validation layers: + +- native shell-driven GNU Hello build: + - `tests/native-build/run-gnu-hello.sh` +- Scheme-driven GNU Hello builder-phase prototype: + - `tests/native-build/gnu-hello-guix-phase-runner.scm` + - `tests/native-build/run-gnu-hello-guix-phase-runner.sh` +- Scheme-driven GNU which builder-phase prototype: + - `tests/native-build/gnu-which-guix-phase-runner.scm` + - `tests/native-build/run-gnu-which-guix-phase-runner.sh` + +Those prototypes prove that FreeBSD can already execute a small but real subset of Guix builder-side GNU build logic when paired with the fixed local Guile build. + +They do **not** yet prove that FreeBSD can perform an actual Guix package build. + +## Current validated surface + +What is already validated on FreeBSD amd64: + +1. Guile can run the necessary Scheme code, provided the locally fixed Guile build is used. +2. Guix builder-side `(guix build gnu-build-system)` code can run a subset of `%standard-phases`. +3. Two small GNU packages build successfully through that subset: + - GNU Hello + - GNU which +4. The following builder phases have been exercised successfully: + - `set-SOURCE-DATE-EPOCH` + - `unpack` + - `configure` + - `build` + - `check` + - `install` + +This means the question is no longer "can FreeBSD run any Guix builder-side GNU phase logic at all?" + +The question is now "what still separates these prototypes from a real Guix package build?" + +## What a real Guix package build does beyond the current prototypes + +At a high level, a real Guix package build goes through these layers: + +1. package object -> bag lowering + - `package->bag` in `~/repos/guix/guix/packages.scm` +2. bag -> derivation lowering + - `bag->derivation` in `~/repos/guix/guix/packages.scm` +3. build-system lowering and builder gexp creation + - `lower` and `gnu-build` in `~/repos/guix/guix/build-system/gnu.scm` +4. gexp residualization into store items and imported module closures + - `lower-object` in `~/repos/guix/guix/gexp.scm` +5. daemon connection and build request submission + - `open-connection` and `set-build-options` in `~/repos/guix/guix/store.scm` + - `build-derivations` in `~/repos/guix/guix/derivations.scm` +6. daemon-side execution with store management, build users, and isolation + - `guix-daemon` behavior documented in `~/repos/guix/doc/guix.texi` + - daemon implementation details in `~/repos/guix/nix/libstore/build.cc` + +The current FreeBSD prototypes only validate a slice of step 3, and they do so in a manually constructed environment. + +## Gap matrix + +### 1. Package lowering is not exercised yet + +Current prototypes: + +- call `gnu-build` directly +- manually provide the source tarball URL/hash context +- manually choose outputs and phases + +Missing real Guix behavior: + +- no `` object is lowered through `package->bag` +- no implicit inputs are attached by `lower` +- no `package->derivation` call occurs +- no real `.drv` file is generated + +Why this matters: + +A large amount of actual Guix behavior lives above the builder-phase layer. The package record, arguments, native inputs, propagated inputs, outputs, search paths, and implicit inputs are all resolved before the builder runs. + +FreeBSD implication: + +The current prototypes show that builder-side Scheme logic works, but they do not yet show that Guix's host-side package lowering logic can successfully target FreeBSD. + +### 2. Implicit GNU build-system inputs are bypassed + +In real Guix, `gnu-build-system` adds implicit inputs through `standard-packages` in `~/repos/guix/guix/build-system/gnu.scm`, which ultimately comes from `%final-inputs` in `~/repos/guix/gnu/packages/commencement.scm`. + +That implicit input set includes toolchain and utility packages such as: + +- `tar` +- `gzip` +- `bzip2` +- `file` +- `diffutils` +- `patch` +- `findutils` +- `gawk` +- `zstd` +- `sed` +- `grep` +- `xz` +- `coreutils` +- `make` +- `bash` +- `ld-wrapper` +- `binutils` +- `gcc` +- `libc` +- `libc:static` + +Current prototypes instead rely on host tools already present on FreeBSD, such as: + +- `fetch` +- `sha256` +- `bsdtar` +- `/bin/sh` +- `/usr/bin/install` +- host `cc` +- host `make` + +Why this matters: + +A real Guix package build is not just "run the phases somehow". It runs them with a specific Guix-managed tool universe and search-path model. + +FreeBSD implication: + +The current prototypes do not yet answer whether Guix's implicit input model can be lowered, realized, and used coherently on FreeBSD. + +### 3. Imported module closure and build-side module materialization are bypassed + +Real `gnu-build` in `~/repos/guix/guix/build-system/gnu.scm` builds a gexp that uses `with-imported-modules` and `gexp->derivation` so that builder-side modules become store material. + +Current prototypes instead: + +- set `GUILE_LOAD_PATH` directly to `~/repos/guix` +- load Guix modules straight from the checkout +- do not build/store an imported module closure +- do not validate generated Guix configuration modules such as `guix config` + +Why this matters: + +This is one of the biggest remaining gaps. Our prototypes prove that the builder-side code itself can run, but they do not prove that Guix can correctly package up the builder program, its modules, and its imported dependencies as a derivation input graph. + +FreeBSD implication: + +Before a true Guix package build can work, FreeBSD will need a functioning path for: + +- imported Guix Scheme modules +- build-side module resolution in a store context +- derivation generation for those module closures + +### 4. The store is still only simulated + +Current prototypes use store-like output names under `/tmp`, for example: + +- `/tmp/.../0000000000000000-hello-2.12.3` +- `/tmp/.../0000000000000000-which-2.21` + +Missing real Guix behavior: + +- no canonical `/gnu/store` +- no true content-addressed store path computation +- no `.drv` path registration +- no store database updates +- no garbage-collection roots +- no validity registration or reference graph maintenance + +Why this matters: + +Guix's reproducibility, dependency tracking, and garbage collection all depend on the real store, not just on an output directory that looks store-like. + +FreeBSD implication: + +A real FreeBSD port must eventually establish a functioning `/gnu/store` equivalent with daemon-managed metadata, not just staged output directories. + +### 5. No daemon RPC path has been validated + +Real Guix clients talk to the daemon using `open-connection` in `~/repos/guix/guix/store.scm` and submit builds through `build-derivations` in `~/repos/guix/guix/derivations.scm`. + +Current prototypes: + +- do not connect to a daemon +- do not negotiate the daemon protocol +- do not set build options via `set-build-options` +- do not submit derivations for building + +Why this matters: + +Even if derivation generation were to succeed on FreeBSD, Guix still needs an operational daemon path to realize builds and manage the store. + +FreeBSD implication: + +At some point the port must stop being "Scheme code can run locally" and become "a Guix client can ask a daemon to build something". + +### 6. No real build isolation or build-user model is in place + +The Guix manual section on `guix-daemon` documents that builds normally run: + +- under dedicated build users from `--build-users-group` +- in a chroot containing only declared dependencies +- and on GNU/Linux, inside additional namespaces/containers + +The daemon code in `~/repos/guix/nix/libstore/build.cc` also clearly depends on: + +- a build-users group +- UID switching +- temporary build directories +- chroot setup +- Linux container features when available + +Current prototypes: + +- run as the current user +- do not switch to build users +- do not restrict filesystem visibility to declared inputs +- do not use chroot +- do not use jails +- do not isolate network access + +Why this matters: + +This is the largest architectural gap relative to the final FreeBSD goal. + +FreeBSD implication: + +The current results are strong evidence for builder-side portability, but they say almost nothing yet about the eventual jail/build-user/isolation design required for a proper port. + +### 7. Substitutes, grafts, and offloading are untouched + +Real Guix package builds may involve: + +- substitutes +- grafts +- daemon-side build options +- offloading + +Relevant code paths appear in: + +- `package->derivation` in `~/repos/guix/guix/packages.scm` +- `set-build-options` in `~/repos/guix/guix/store.scm` +- `guix-daemon` documentation in `~/repos/guix/doc/guix.texi` + +Current prototypes: + +- always build locally +- do not use substitutes +- do not test graft behavior +- do not exercise offload logic + +Why this matters: + +These are not the first blockers for a minimal FreeBSD prototype, but they remain part of the full Guix execution model. + +## Practical conclusion + +The current FreeBSD work has successfully de-risked the following narrow question: + +> Can FreeBSD run Guix builder-side GNU build phases for small packages when Guile is fixed? + +Answer: yes. + +The next blockers are now clearly above that layer: + +1. package lowering to bags/derivations +2. imported module/store materialization +3. real store and daemon connectivity +4. daemon-side isolation and build-user execution + +## Most actionable next milestone + +The smallest high-value milestone after this analysis is likely: + +### Minimal milestone: derivation-generation validation without claiming full build support + +Try to validate that FreeBSD can at least get through the host-side lowering path for a tiny package far enough to produce a derivation or identify the first concrete failure point. + +That would answer a much sharper question than the current prototypes: + +> Does FreeBSD fail first in host-side Guix lowering, in store interaction, or in daemon execution? + +This is the narrowest next step that meaningfully approaches a real Guix package build. + +## Recommended next step + +Proceed with a targeted derivation-generation investigation for a tiny package, using the current GNU Hello package as the first candidate if practical, and record the exact first failure boundary among: + +- missing generated Guix modules/configuration +- store connection assumptions +- derivation emission +- daemon availability +- daemon-side execution assumptions