From 47d31e8992726e52aa49ef33fe1e1320bc966549 Mon Sep 17 00:00:00 2001 From: Steffen Beyer Date: Wed, 1 Apr 2026 10:48:54 +0200 Subject: [PATCH] Build local Guile-JSON on FreeBSD --- docs/PROGRESS.md | 66 ++++++ docs/reports/phase1-guile-json-freebsd.md | 181 ++++++++++++++++ tests/guix/build-local-guile-json.sh | 194 ++++++++++++++++++ ...run-derivation-generation-investigation.sh | 27 ++- 4 files changed, 467 insertions(+), 1 deletion(-) create mode 100644 docs/reports/phase1-guile-json-freebsd.md create mode 100755 tests/guix/build-local-guile-json.sh diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index c232904..49e0107 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -571,3 +571,69 @@ Next recommended step: 1. obtain `Guile-JSON` compatible with the fixed local Guile build and install it into the same local dependency prefix 2. re-run the derivation-generation investigation again to identify the next configure-time or checkout-time blocker after `Guile-JSON` 3. continue keeping `/frx/store` as the intended experimental store root and keep `~/repos/bdwgc` in reserve if later FreeBSD-specific GC/thread issues appear + +## 2026-04-01 — Phase 1.2 follow-up: local Guile-JSON built on FreeBSD; next blocker is Guile-SQLite3 + +Completed work: + +- added a reproducible local Guile-JSON build harness: + - `tests/guix/build-local-guile-json.sh` +- updated the derivation-generation investigation harness to probe and record local recent-enough `(json)` availability in addition to the earlier `(gnutls)` and `(git)` checks +- used the current Guix package definition in `~/repos/guix/gnu/packages/guile.scm` as the source of truth for: + - `guile-json` version `4.7.3` + - expected Guix nix-base32 source hash `127k2xc07w1gnyqs40z4865l8p3ra5xgpcn569dz04lxsa709fiq` +- verified the downloaded Guile-JSON tarball against the translated SHA256: + - `38ba048ed29d12f05b32c5b2fb7a51795c448b41e403a2b1b72ff0035817f388` +- built and installed a local Guile-JSON validation copy into the same local dependency prefix already used for checkout prerequisites: + - `/tmp/guile-gnutls-freebsd-validate-install` +- validated successfully that the fixed local Guile now satisfies the Guix configure-time JSON requirement by: + - loading `(json)` + - using `define-json-mapping` + - decoding a small JSON object successfully +- re-ran the checkout derivation-generation investigation with: + - `GUILE_EXTRA_PREFIX=/tmp/guile-gnutls-freebsd-validate-install` + - store directory still set to `/frx/store` +- wrote the results to: + - `docs/reports/phase1-guile-json-freebsd.md` + +Important findings: + +- unlike the earlier Guile-Git step, Guile-JSON built cleanly from the release tarball and did not require autotools regeneration in this validation pass +- unlike the earlier Guile-GnuTLS step, no FreeBSD-specific source patch was needed here +- the shared local validation prefix now contains at least: + - Guile-GnuTLS + - Guile bytestructures + - Guile-Git + - Guile-JSON +- the Guix checkout now gets past the previously cleared dependency gates for: + - `(gnutls)` + - `Guile-Git` + - `Guile-JSON` +- after clearing those, the next configure-time blocker is now: + - `configure: error: A recent Guile-SQLite3 could not be found; please install it.` + +Current assessment: + +- the checkout-preparation path on FreeBSD has progressed another dependency layer deeper +- the project still has not yet reached derivation emission, daemon connectivity, or actual `/frx/store` population +- the next concrete blocker is now recent `Guile-SQLite3`, as reported directly by the real Guix checkout configure step + +Recent commits: + +- `e380e88` — `Add FreeBSD Guile verification harness` +- `cd721b1` — `Update progress after Guile verification` +- `27916cb` — `Diagnose Guile subprocess crash on FreeBSD` +- `02f7a7f` — `Validate local Guile fix on FreeBSD` +- `4aebea4` — `Add native GNU Hello FreeBSD build harness` +- `c944cdb` — `Validate Guix builder phases on FreeBSD` +- `0a2e48e` — `Validate GNU which builder phases on FreeBSD` +- `245a47d` — `Document gaps to real Guix FreeBSD builds` +- `d62e9b0` — `Investigate Guix derivation generation on FreeBSD` +- `c0a85ed` — `Build local Guile-GnuTLS on FreeBSD` +- `15b9037` — `Build local Guile-Git on FreeBSD` + +Next recommended step: + +1. obtain recent `Guile-SQLite3` compatible with the fixed local Guile build and install it into the same local dependency prefix +2. re-run the derivation-generation investigation again to identify the next configure-time or checkout-time blocker after `Guile-SQLite3` +3. continue keeping `/frx/store` as the intended experimental store root and keep `~/repos/bdwgc` in reserve if later FreeBSD-specific GC/thread issues appear diff --git a/docs/reports/phase1-guile-json-freebsd.md b/docs/reports/phase1-guile-json-freebsd.md new file mode 100644 index 0000000..88a5b47 --- /dev/null +++ b/docs/reports/phase1-guile-json-freebsd.md @@ -0,0 +1,181 @@ +# Phase 1.2 follow-up: local Guile-JSON built on FreeBSD and checkout investigation advanced again + +Date: 2026-04-01 + +## Summary + +This step addressed the next Guix checkout blocker found after adding local Guile-GnuTLS and Guile-Git support on FreeBSD. + +Added/updated files: + +- `tests/guix/build-local-guile-json.sh` +- updated `tests/guix/run-derivation-generation-investigation.sh` + +The result is: + +1. `guile-json` can be built successfully on FreeBSD against the previously validated fixed local Guile build +2. it can be installed into the same local dependency prefix already used for the earlier checkout prerequisite work +3. after making `(json)` available, the Guix checkout configure step advances again +4. the next checkout blocker is now `Guile-SQLite3` + +## Inputs used + +### Local Guile + +The same fixed local Guile build was used: + +- `/tmp/guile-freebsd-validate-install/bin/guile` + +### Existing local dependency prefix + +This step reused the same local dependency prefix already populated by earlier prerequisite builds: + +- `/tmp/guile-gnutls-freebsd-validate-install` + +At this point that prefix contains at least: + +- Guile-GnuTLS +- Guile bytestructures +- Guile-Git +- Guile-JSON + +### Guix source-of-truth package definition + +This step used the current Guix package definition in `~/repos/guix/gnu/packages/guile.scm`: + +- `guile-json` +- version `4.7.3` +- Guix nix-base32 source hash: + - `127k2xc07w1gnyqs40z4865l8p3ra5xgpcn569dz04lxsa709fiq` + +Translated and verified SHA256: + +```text +38ba048ed29d12f05b32c5b2fb7a51795c448b41e403a2b1b72ff0035817f388 +``` + +## Build findings + +### 1. The release tarball builds cleanly on FreeBSD with the fixed local Guile + +Unlike the earlier Guile-Git step, `guile-json` was built from the upstream release tarball rather than a Git checkout, so no autotools regeneration step was required in this validation pass. + +### 2. No new FreeBSD-specific source patch was needed + +Unlike the earlier Guile-GnuTLS step, this build did not require any FreeBSD-specific source patch. + +### 3. The validation matched Guix's actual configure-time expectation + +Guix does not merely check that `(json)` loads; `configure.ac` calls `GUIX_CHECK_GUILE_JSON`, which requires a recent-enough Guile-JSON providing `define-json-mapping`. + +The local validation therefore used the same kind of test: + +- load `(json)` +- define a small JSON mapping with `define-json-mapping` +- decode a JSON object and confirm the mapped record matches the expected value + +Observed validation result: + +```text +ok +``` + +## Validation command + +```sh +ENV_OUT=/tmp/guile-json-env.sh \ +METADATA_OUT=/tmp/guile-json-metadata.txt \ +./tests/guix/build-local-guile-json.sh +``` + +Default install prefix used by the harness: + +```text +/tmp/guile-gnutls-freebsd-validate-install +``` + +## Result + +The local Guile-JSON build succeeded. + +Observed checks: + +```text +json module check: ok +Existing (gnutls) module in prefix: present +Existing (git) module in prefix: present +``` + +This means the same local prefix now satisfies at least the following mandatory Guix checkout module requirements previously encountered on FreeBSD: + +- `(gnutls)` +- `(git)` with recent Guile-Git export support +- `(json)` with recent-enough `define-json-mapping` support + +## Re-running the checkout investigation + +After installing Guile-JSON, the derivation-generation investigation was re-run with: + +```sh +GUILE_EXTRA_PREFIX=/tmp/guile-gnutls-freebsd-validate-install \ +METADATA_OUT=/tmp/guix-derivation-investigation-with-json.txt \ +./tests/guix/run-derivation-generation-investigation.sh +``` + +The investigation continued to use the requested experimental directories: + +- store: `/frx/store` +- local state: `/frx/var` +- sysconf: `/frx/etc` + +## Updated result + +With `(gnutls)`, `(git)`, and `(json)` available, the Guix checkout configure step progresses farther and now fails at the next missing Guile dependency: + +```text +configure: error: A recent Guile-SQLite3 could not be found; please install it. +``` + +Relevant configure summary lines observed: + +```text +checking whether Guile-JSON is available and recent enough... yes +configure: error: A recent Guile-SQLite3 could not be found; please install it. +``` + +## What this step demonstrates + +This step demonstrates that: + +1. the earlier `Guile-JSON` blocker is genuinely cleared on FreeBSD when using the fixed local Guile build plus the local dependency prefix +2. the checkout can move one more dependency layer deeper than before +3. the project is still in the checkout-prerequisite stage rather than the derivation/store/daemon stage +4. the next concrete prerequisite for reaching deeper Guix checkout usability on FreeBSD is `Guile-SQLite3` + +## Current blocker stack + +The current checkout path is now blocked by: + +1. unsupported-platform configure gating unless `--with-courage` is used +2. missing recent `Guile-SQLite3` after supplying: + - Guile-GnuTLS + - Guile-Git + - Guile-JSON + +The earlier missing blockers have now been cleared locally for validation: + +- `(gnutls)` +- `(git)` / recent Guile-Git export requirement +- recent `(json)` support + +## Recommended next step + +Obtain or build `Guile-SQLite3` compatible with the same fixed local Guile build, install it into the same local dependency prefix, and re-run the derivation-generation investigation again. + +That should show whether the next boundary lies at: + +- `Guile-Gcrypt` +- `Guile-zlib` +- `Guile-lzlib` +- `Guile-semver` +- or finally something beyond configure-time Guile module prerequisites diff --git a/tests/guix/build-local-guile-json.sh b/tests/guix/build-local-guile-json.sh new file mode 100755 index 0000000..faa9382 --- /dev/null +++ b/tests/guix/build-local-guile-json.sh @@ -0,0 +1,194 @@ +#!/bin/sh +set -eu + +source_url=${GUILE_JSON_SOURCE_URL:-"https://download.savannah.gnu.org/releases/guile-json/guile-json-4.7.3.tar.gz"} +version=${GUILE_JSON_VERSION:-4.7.3} +expected_nix_base32=${GUILE_JSON_NIX_BASE32:-127k2xc07w1gnyqs40z4865l8p3ra5xgpcn569dz04lxsa709fiq} +install_prefix=${INSTALL_PREFIX:-/tmp/guile-gnutls-freebsd-validate-install} +guile_bin=${GUILE_BIN:-/tmp/guile-freebsd-validate-install/bin/guile} +guix_source_dir=${GUIX_SOURCE_DIR:-"$HOME/repos/guix"} +make_bin=${MAKE_BIN:-gmake} + +if [ ! -x "$guile_bin" ]; then + echo "Guile binary is not executable: $guile_bin" >&2 + exit 1 +fi +if [ ! -d "$guix_source_dir/guix" ]; then + echo "Guix source tree not found at $guix_source_dir" >&2 + exit 1 +fi +for tool in fetch sha256 bsdtar "$make_bin"; do + if ! command -v "$tool" >/dev/null 2>&1; then + echo "Required tool not found: $tool" >&2 + exit 1 + fi +done + +cleanup=0 +if [ -n "${WORKDIR:-}" ]; then + workdir=$WORKDIR + mkdir -p "$workdir" +else + workdir=$(mktemp -d /tmp/fruix-guile-json.XXXXXX) + cleanup=1 +fi + +if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then + cleanup=0 +fi + +cleanup_workdir() { + if [ "$cleanup" -eq 1 ]; then + rm -rf "$workdir" + fi +} +trap cleanup_workdir EXIT INT TERM + +guile_bindir=$(CDPATH= cd -- "$(dirname "$guile_bin")" && pwd) +guile_prefix=$(CDPATH= cd -- "$guile_bindir/.." && pwd) +guile_lib_dir=$guile_prefix/lib +guile_version=$(LD_LIBRARY_PATH="$guile_lib_dir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" "$guile_bin" -c '(display (effective-version))') +site_dir=$install_prefix/share/guile/site/$guile_version +site_ccache_dir=$install_prefix/lib/guile/$guile_version/site-ccache +extensions_dir=$install_prefix/lib/guile/$guile_version/extensions + +tool_bindir=$workdir/guile-tools-bin +mkdir -p "$tool_bindir" +ln -sf "$guile_bin" "$tool_bindir/guile-3.0" +ln -sf "$guile_bin" "$tool_bindir/guile" +ln -sf "$guile_bindir/guild" "$tool_bindir/guild-3.0" +ln -sf "$guile_bindir/guild" "$tool_bindir/guild" +ln -sf "$guile_bindir/guile-config" "$tool_bindir/guile-config-3.0" +ln -sf "$guile_bindir/guile-config" "$tool_bindir/guile-config" +ln -sf "$guile_bindir/guile-snarf" "$tool_bindir/guile-snarf" +export PATH="$tool_bindir:$guile_bindir:/usr/local/bin:$PATH" +export ACLOCAL_PATH=/usr/local/share/aclocal${ACLOCAL_PATH:+:$ACLOCAL_PATH} +export PKG_CONFIG_PATH=/usr/local/libdata/pkgconfig:/usr/local/lib/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH} +export CPPFLAGS='-I/usr/local/include' +export LDFLAGS="-L/usr/local/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,$guile_lib_dir -Wl,-rpath,$install_prefix/lib" +if [ -n "${LD_LIBRARY_PATH:-}" ]; then + export LD_LIBRARY_PATH="$install_prefix/lib:$guile_lib_dir:/usr/local/lib:$LD_LIBRARY_PATH" +else + export LD_LIBRARY_PATH="$install_prefix/lib:$guile_lib_dir:/usr/local/lib" +fi +if [ -d "$site_dir" ]; then + export GUILE_LOAD_PATH="$site_dir${GUILE_LOAD_PATH:+:$GUILE_LOAD_PATH}" +fi +if [ -d "$site_ccache_dir" ]; then + export GUILE_LOAD_COMPILED_PATH="$site_ccache_dir${GUILE_LOAD_COMPILED_PATH:+:$GUILE_LOAD_COMPILED_PATH}" +fi +if [ -d "$extensions_dir" ]; then + export GUILE_EXTENSIONS_PATH="$extensions_dir${GUILE_EXTENSIONS_PATH:+:$GUILE_EXTENSIONS_PATH}" +fi + +src_tarball=$workdir/guile-json-$version.tar.gz +src_dir=$workdir/guile-json-$version +build_dir=$workdir/build +metadata_file=$workdir/guile-json-build-metadata.txt +env_file=$workdir/guile-json-env.sh +configure_log=$workdir/configure.log +build_log=$workdir/build.log +install_log=$workdir/install.log + +expected_sha256_hex=$(GUILE_AUTO_COMPILE=0 \ + GUILE_LOAD_PATH="$guix_source_dir${GUILE_LOAD_PATH:+:$GUILE_LOAD_PATH}" \ + "$guile_bin" -c "(use-modules (guix base32) (rnrs bytevectors) (ice-9 format)) (for-each (lambda (b) (format #t \"~2,'0x\" b)) (bytevector->u8-list (nix-base32-string->bytevector (cadr (command-line))))) (newline)" \ + "$expected_nix_base32") + +printf 'Using Guile: %s\n' "$guile_bin" +printf 'Installing into existing prefix: %s\n' "$install_prefix" +printf 'Working directory: %s\n' "$workdir" +printf 'Fetching: %s\n' "$source_url" +fetch -o "$src_tarball" "$source_url" +actual_sha256_hex=$(sha256 -q "$src_tarball") +if [ "$actual_sha256_hex" != "$expected_sha256_hex" ]; then + echo "sha256 mismatch for $src_tarball" >&2 + echo "expected: $expected_sha256_hex" >&2 + echo "actual: $actual_sha256_hex" >&2 + exit 1 +fi + +bsdtar -xf "$src_tarball" -C "$workdir" +if [ ! -d "$src_dir" ]; then + echo "expected extracted source directory not found: $src_dir" >&2 + exit 1 +fi + +rm -rf "$site_dir/json" "$site_dir/json.scm" "$site_ccache_dir/json" "$site_ccache_dir/json.go" +mkdir -p "$build_dir" +( + cd "$build_dir" + "$src_dir/configure" --prefix="$install_prefix" +) >"$configure_log" 2>&1 +( + cd "$build_dir" + "$make_bin" GUILE_AUTO_COMPILE=0 -j"${JOBS:-$(sysctl -n hw.ncpu 2>/dev/null || echo 1)}" +) >"$build_log" 2>&1 +( + cd "$build_dir" + "$make_bin" GUILE_AUTO_COMPILE=0 install +) >"$install_log" 2>&1 + +export GUILE_LOAD_PATH="$site_dir${GUILE_LOAD_PATH:+:$GUILE_LOAD_PATH}" +export GUILE_LOAD_COMPILED_PATH="$site_ccache_dir${GUILE_LOAD_COMPILED_PATH:+:$GUILE_LOAD_COMPILED_PATH}" +json_check=$("$guile_bin" -c '(use-modules (json)) (define-json-mapping make-frob frob? json->frob (a frob-a) (b frob-b "bee")) (display (if (equal? (json->frob (open-input-string "{ \"a\": 1, \"bee\": 2 }")) (make-frob 1 2)) "ok" "mismatch")) (newline)') +if [ "$json_check" != "ok" ]; then + echo "(json) module validation failed" >&2 + exit 1 +fi + +gnutls_check=$("$guile_bin" -c '(catch #t (lambda () (use-modules (gnutls)) (display "present") (newline)) (lambda _ (display "missing") (newline)))') +git_check=$("$guile_bin" -c '(catch #t (lambda () (use-modules (git)) (display (if (procedure? graph-descendant?) "present" "missing-export")) (newline)) (lambda _ (display "missing") (newline)))') + +cat >"$env_file" <"$metadata_file" < make-frob frob? json->frob (a frob-a) (b frob-b "bee")) (display (if (equal? (json->frob (open-input-string "{ \"a\": 1, \"bee\": 2 }")) (make-frob 1 2)) "ok" "mismatch")) (newline)' >"$json_check_out" 2>"$json_check_err" +check_rc=$? +set -e +if [ "$check_rc" -eq 0 ] && [ "$(tr -d '\n' < "$json_check_out")" = ok ]; then + local_json_check=present +fi + if [ "$unsupported_rc" -eq 0 ]; then echo "configure without --with-courage unexpectedly succeeded" >&2 exit 1 @@ -170,6 +181,17 @@ if [ "$local_gnutls_check" = present ] && [ "$local_git_check" = missing ]; then exit 1 fi fi +if [ "$local_gnutls_check" = present ] && [ "$local_git_check" = present ] && [ "$local_json_check" = missing ]; then + if ! grep -q "Guile-JSON is missing" "$courage_log"; then + echo "configure with --with-courage failed, but not for the expected missing-Guile-JSON reason" >&2 + exit 1 + fi +elif [ "$local_json_check" = present ]; then + if grep -q "Guile-JSON is missing" "$courage_log"; then + echo "configure with --with-courage still reports missing Guile-JSON even though (json) works" >&2 + exit 1 + fi +fi cat >"$metadata_file" <} local_guile_gnutls_module=$local_gnutls_check local_guile_git_module=$local_git_check +local_guile_json_module=$local_json_check unsupported_summary_begin $unsupported_summary unsupported_summary_end @@ -208,8 +231,10 @@ if [ "$local_gnutls_check" = missing ]; then printf 'Finding 2: configure with --with-courage is currently blocked by missing Guile (gnutls) bindings.\n' elif [ "$local_git_check" = missing ]; then printf 'Finding 2: Guile (gnutls) is available, but configure is now blocked by missing Guile-Git.\n' +elif [ "$local_json_check" = missing ]; then + printf 'Finding 2: Guile (gnutls) and Guile-Git are available, but configure is now blocked by missing Guile-JSON.\n' else - printf 'Finding 2: Guile (gnutls) and Guile-Git are available; next blocker: %s\n' "${courage_error:-unknown configure failure}" + printf 'Finding 2: Guile (gnutls), Guile-Git, and Guile-JSON are available; next blocker: %s\n' "${courage_error:-unknown configure failure}" fi if [ "$cleanup" -eq 0 ]; then printf 'Metadata file: %s\n' "$metadata_file"