Build local Guile-JSON on FreeBSD

2026-04-01 10:48:54 +02:00
parent 15b9037034
commit 47d31e8992
4 changed files with 467 additions and 1 deletions
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -571,3 +571,69 @@ Next recommended step:
 1. obtain `Guile-JSON` compatible with the fixed local Guile build and install it into the same local dependency prefix
 2. re-run the derivation-generation investigation again to identify the next configure-time or checkout-time blocker after `Guile-JSON`
 3. continue keeping `/frx/store` as the intended experimental store root and keep `~/repos/bdwgc` in reserve if later FreeBSD-specific GC/thread issues appear
+
+## 2026-04-01 — Phase 1.2 follow-up: local Guile-JSON built on FreeBSD; next blocker is Guile-SQLite3
+
+Completed work:
+
+- added a reproducible local Guile-JSON build harness:
+  - `tests/guix/build-local-guile-json.sh`
+- updated the derivation-generation investigation harness to probe and record local recent-enough `(json)` availability in addition to the earlier `(gnutls)` and `(git)` checks
+- used the current Guix package definition in `~/repos/guix/gnu/packages/guile.scm` as the source of truth for:
+  - `guile-json` version `4.7.3`
+  - expected Guix nix-base32 source hash `127k2xc07w1gnyqs40z4865l8p3ra5xgpcn569dz04lxsa709fiq`
+- verified the downloaded Guile-JSON tarball against the translated SHA256:
+  - `38ba048ed29d12f05b32c5b2fb7a51795c448b41e403a2b1b72ff0035817f388`
+- built and installed a local Guile-JSON validation copy into the same local dependency prefix already used for checkout prerequisites:
+  - `/tmp/guile-gnutls-freebsd-validate-install`
+- validated successfully that the fixed local Guile now satisfies the Guix configure-time JSON requirement by:
+  - loading `(json)`
+  - using `define-json-mapping`
+  - decoding a small JSON object successfully
+- re-ran the checkout derivation-generation investigation with:
+  - `GUILE_EXTRA_PREFIX=/tmp/guile-gnutls-freebsd-validate-install`
+  - store directory still set to `/frx/store`
+- wrote the results to:
+  - `docs/reports/phase1-guile-json-freebsd.md`
+
+Important findings:
+
+- unlike the earlier Guile-Git step, Guile-JSON built cleanly from the release tarball and did not require autotools regeneration in this validation pass
+- unlike the earlier Guile-GnuTLS step, no FreeBSD-specific source patch was needed here
+- the shared local validation prefix now contains at least:
+  - Guile-GnuTLS
+  - Guile bytestructures
+  - Guile-Git
+  - Guile-JSON
+- the Guix checkout now gets past the previously cleared dependency gates for:
+  - `(gnutls)`
+  - `Guile-Git`
+  - `Guile-JSON`
+- after clearing those, the next configure-time blocker is now:
+  - `configure: error: A recent Guile-SQLite3 could not be found; please install it.`
+
+Current assessment:
+
+- the checkout-preparation path on FreeBSD has progressed another dependency layer deeper
+- the project still has not yet reached derivation emission, daemon connectivity, or actual `/frx/store` population
+- the next concrete blocker is now recent `Guile-SQLite3`, as reported directly by the real Guix checkout configure step
+
+Recent commits:
+
+- `e380e88` — `Add FreeBSD Guile verification harness`
+- `cd721b1` — `Update progress after Guile verification`
+- `27916cb` — `Diagnose Guile subprocess crash on FreeBSD`
+- `02f7a7f` — `Validate local Guile fix on FreeBSD`
+- `4aebea4` — `Add native GNU Hello FreeBSD build harness`
+- `c944cdb` — `Validate Guix builder phases on FreeBSD`
+- `0a2e48e` — `Validate GNU which builder phases on FreeBSD`
+- `245a47d` — `Document gaps to real Guix FreeBSD builds`
+- `d62e9b0` — `Investigate Guix derivation generation on FreeBSD`
+- `c0a85ed` — `Build local Guile-GnuTLS on FreeBSD`
+- `15b9037` — `Build local Guile-Git on FreeBSD`
+
+Next recommended step:
+
+1. obtain recent `Guile-SQLite3` compatible with the fixed local Guile build and install it into the same local dependency prefix
+2. re-run the derivation-generation investigation again to identify the next configure-time or checkout-time blocker after `Guile-SQLite3`
+3. continue keeping `/frx/store` as the intended experimental store root and keep `~/repos/bdwgc` in reserve if later FreeBSD-specific GC/thread issues appear
--- a/docs/reports/phase1-guile-json-freebsd.md
+++ b/docs/reports/phase1-guile-json-freebsd.md
@@ -0,0 +1,181 @@
+# Phase 1.2 follow-up: local Guile-JSON built on FreeBSD and checkout investigation advanced again
+
+Date: 2026-04-01
+
+## Summary
+
+This step addressed the next Guix checkout blocker found after adding local Guile-GnuTLS and Guile-Git support on FreeBSD.
+
+Added/updated files:
+
+- `tests/guix/build-local-guile-json.sh`
+- updated `tests/guix/run-derivation-generation-investigation.sh`
+
+The result is:
+
+1. `guile-json` can be built successfully on FreeBSD against the previously validated fixed local Guile build
+2. it can be installed into the same local dependency prefix already used for the earlier checkout prerequisite work
+3. after making `(json)` available, the Guix checkout configure step advances again
+4. the next checkout blocker is now `Guile-SQLite3`
+
+## Inputs used
+
+### Local Guile
+
+The same fixed local Guile build was used:
+
+- `/tmp/guile-freebsd-validate-install/bin/guile`
+
+### Existing local dependency prefix
+
+This step reused the same local dependency prefix already populated by earlier prerequisite builds:
+
+- `/tmp/guile-gnutls-freebsd-validate-install`
+
+At this point that prefix contains at least:
+
+- Guile-GnuTLS
+- Guile bytestructures
+- Guile-Git
+- Guile-JSON
+
+### Guix source-of-truth package definition
+
+This step used the current Guix package definition in `~/repos/guix/gnu/packages/guile.scm`:
+
+- `guile-json`
+- version `4.7.3`
+- Guix nix-base32 source hash:
+  - `127k2xc07w1gnyqs40z4865l8p3ra5xgpcn569dz04lxsa709fiq`
+
+Translated and verified SHA256:
+
+```text
+38ba048ed29d12f05b32c5b2fb7a51795c448b41e403a2b1b72ff0035817f388
+```
+
+## Build findings
+
+### 1. The release tarball builds cleanly on FreeBSD with the fixed local Guile
+
+Unlike the earlier Guile-Git step, `guile-json` was built from the upstream release tarball rather than a Git checkout, so no autotools regeneration step was required in this validation pass.
+
+### 2. No new FreeBSD-specific source patch was needed
+
+Unlike the earlier Guile-GnuTLS step, this build did not require any FreeBSD-specific source patch.
+
+### 3. The validation matched Guix's actual configure-time expectation
+
+Guix does not merely check that `(json)` loads; `configure.ac` calls `GUIX_CHECK_GUILE_JSON`, which requires a recent-enough Guile-JSON providing `define-json-mapping`.
+
+The local validation therefore used the same kind of test:
+
+- load `(json)`
+- define a small JSON mapping with `define-json-mapping`
+- decode a JSON object and confirm the mapped record matches the expected value
+
+Observed validation result:
+
+```text
+ok
+```
+
+## Validation command
+
+```sh
+ENV_OUT=/tmp/guile-json-env.sh \
+METADATA_OUT=/tmp/guile-json-metadata.txt \
+./tests/guix/build-local-guile-json.sh
+```
+
+Default install prefix used by the harness:
+
+```text
+/tmp/guile-gnutls-freebsd-validate-install
+```
+
+## Result
+
+The local Guile-JSON build succeeded.
+
+Observed checks:
+
+```text
+json module check: ok
+Existing (gnutls) module in prefix: present
+Existing (git) module in prefix: present
+```
+
+This means the same local prefix now satisfies at least the following mandatory Guix checkout module requirements previously encountered on FreeBSD:
+
+- `(gnutls)`
+- `(git)` with recent Guile-Git export support
+- `(json)` with recent-enough `define-json-mapping` support
+
+## Re-running the checkout investigation
+
+After installing Guile-JSON, the derivation-generation investigation was re-run with:
+
+```sh
+GUILE_EXTRA_PREFIX=/tmp/guile-gnutls-freebsd-validate-install \
+METADATA_OUT=/tmp/guix-derivation-investigation-with-json.txt \
+./tests/guix/run-derivation-generation-investigation.sh
+```
+
+The investigation continued to use the requested experimental directories:
+
+- store: `/frx/store`
+- local state: `/frx/var`
+- sysconf: `/frx/etc`
+
+## Updated result
+
+With `(gnutls)`, `(git)`, and `(json)` available, the Guix checkout configure step progresses farther and now fails at the next missing Guile dependency:
+
+```text
+configure: error: A recent Guile-SQLite3 could not be found; please install it.
+```
+
+Relevant configure summary lines observed:
+
+```text
+checking whether Guile-JSON is available and recent enough... yes
+configure: error: A recent Guile-SQLite3 could not be found; please install it.
+```
+
+## What this step demonstrates
+
+This step demonstrates that:
+
+1. the earlier `Guile-JSON` blocker is genuinely cleared on FreeBSD when using the fixed local Guile build plus the local dependency prefix
+2. the checkout can move one more dependency layer deeper than before
+3. the project is still in the checkout-prerequisite stage rather than the derivation/store/daemon stage
+4. the next concrete prerequisite for reaching deeper Guix checkout usability on FreeBSD is `Guile-SQLite3`
+
+## Current blocker stack
+
+The current checkout path is now blocked by:
+
+1. unsupported-platform configure gating unless `--with-courage` is used
+2. missing recent `Guile-SQLite3` after supplying:
+   - Guile-GnuTLS
+   - Guile-Git
+   - Guile-JSON
+
+The earlier missing blockers have now been cleared locally for validation:
+
+- `(gnutls)`
+- `(git)` / recent Guile-Git export requirement
+- recent `(json)` support
+
+## Recommended next step
+
+Obtain or build `Guile-SQLite3` compatible with the same fixed local Guile build, install it into the same local dependency prefix, and re-run the derivation-generation investigation again.
+
+That should show whether the next boundary lies at:
+
+- `Guile-Gcrypt`
+- `Guile-zlib`
+- `Guile-lzlib`
+- `Guile-semver`
+- or finally something beyond configure-time Guile module prerequisites
--- a/tests/guix/build-local-guile-json.sh
+++ b/tests/guix/build-local-guile-json.sh
@@ -0,0 +1,194 @@
+#!/bin/sh
+set -eu
+
+source_url=${GUILE_JSON_SOURCE_URL:-"https://download.savannah.gnu.org/releases/guile-json/guile-json-4.7.3.tar.gz"}
+version=${GUILE_JSON_VERSION:-4.7.3}
+expected_nix_base32=${GUILE_JSON_NIX_BASE32:-127k2xc07w1gnyqs40z4865l8p3ra5xgpcn569dz04lxsa709fiq}
+install_prefix=${INSTALL_PREFIX:-/tmp/guile-gnutls-freebsd-validate-install}
+guile_bin=${GUILE_BIN:-/tmp/guile-freebsd-validate-install/bin/guile}
+guix_source_dir=${GUIX_SOURCE_DIR:-"$HOME/repos/guix"}
+make_bin=${MAKE_BIN:-gmake}
+
+if [ ! -x "$guile_bin" ]; then
+  echo "Guile binary is not executable: $guile_bin" >&2
+  exit 1
+fi
+if [ ! -d "$guix_source_dir/guix" ]; then
+  echo "Guix source tree not found at $guix_source_dir" >&2
+  exit 1
+fi
+for tool in fetch sha256 bsdtar "$make_bin"; do
+  if ! command -v "$tool" >/dev/null 2>&1; then
+    echo "Required tool not found: $tool" >&2
+    exit 1
+  fi
+done
+
+cleanup=0
+if [ -n "${WORKDIR:-}" ]; then
+  workdir=$WORKDIR
+  mkdir -p "$workdir"
+else
+  workdir=$(mktemp -d /tmp/fruix-guile-json.XXXXXX)
+  cleanup=1
+fi
+
+if [ "${KEEP_WORKDIR:-0}" -eq 1 ]; then
+  cleanup=0
+fi
+
+cleanup_workdir() {
+  if [ "$cleanup" -eq 1 ]; then
+    rm -rf "$workdir"
+  fi
+}
+trap cleanup_workdir EXIT INT TERM
+
+guile_bindir=$(CDPATH= cd -- "$(dirname "$guile_bin")" && pwd)
+guile_prefix=$(CDPATH= cd -- "$guile_bindir/.." && pwd)
+guile_lib_dir=$guile_prefix/lib
+guile_version=$(LD_LIBRARY_PATH="$guile_lib_dir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" "$guile_bin" -c '(display (effective-version))')
+site_dir=$install_prefix/share/guile/site/$guile_version
+site_ccache_dir=$install_prefix/lib/guile/$guile_version/site-ccache
+extensions_dir=$install_prefix/lib/guile/$guile_version/extensions
+
+tool_bindir=$workdir/guile-tools-bin
+mkdir -p "$tool_bindir"
+ln -sf "$guile_bin" "$tool_bindir/guile-3.0"
+ln -sf "$guile_bin" "$tool_bindir/guile"
+ln -sf "$guile_bindir/guild" "$tool_bindir/guild-3.0"
+ln -sf "$guile_bindir/guild" "$tool_bindir/guild"
+ln -sf "$guile_bindir/guile-config" "$tool_bindir/guile-config-3.0"
+ln -sf "$guile_bindir/guile-config" "$tool_bindir/guile-config"
+ln -sf "$guile_bindir/guile-snarf" "$tool_bindir/guile-snarf"
+export PATH="$tool_bindir:$guile_bindir:/usr/local/bin:$PATH"
+export ACLOCAL_PATH=/usr/local/share/aclocal${ACLOCAL_PATH:+:$ACLOCAL_PATH}
+export PKG_CONFIG_PATH=/usr/local/libdata/pkgconfig:/usr/local/lib/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}
+export CPPFLAGS='-I/usr/local/include'
+export LDFLAGS="-L/usr/local/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,$guile_lib_dir -Wl,-rpath,$install_prefix/lib"
+if [ -n "${LD_LIBRARY_PATH:-}" ]; then
+  export LD_LIBRARY_PATH="$install_prefix/lib:$guile_lib_dir:/usr/local/lib:$LD_LIBRARY_PATH"
+else
+  export LD_LIBRARY_PATH="$install_prefix/lib:$guile_lib_dir:/usr/local/lib"
+fi
+if [ -d "$site_dir" ]; then
+  export GUILE_LOAD_PATH="$site_dir${GUILE_LOAD_PATH:+:$GUILE_LOAD_PATH}"
+fi
+if [ -d "$site_ccache_dir" ]; then
+  export GUILE_LOAD_COMPILED_PATH="$site_ccache_dir${GUILE_LOAD_COMPILED_PATH:+:$GUILE_LOAD_COMPILED_PATH}"
+fi
+if [ -d "$extensions_dir" ]; then
+  export GUILE_EXTENSIONS_PATH="$extensions_dir${GUILE_EXTENSIONS_PATH:+:$GUILE_EXTENSIONS_PATH}"
+fi
+
+src_tarball=$workdir/guile-json-$version.tar.gz
+src_dir=$workdir/guile-json-$version
+build_dir=$workdir/build
+metadata_file=$workdir/guile-json-build-metadata.txt
+env_file=$workdir/guile-json-env.sh
+configure_log=$workdir/configure.log
+build_log=$workdir/build.log
+install_log=$workdir/install.log
+
+expected_sha256_hex=$(GUILE_AUTO_COMPILE=0 \
+  GUILE_LOAD_PATH="$guix_source_dir${GUILE_LOAD_PATH:+:$GUILE_LOAD_PATH}" \
+  "$guile_bin" -c "(use-modules (guix base32) (rnrs bytevectors) (ice-9 format)) (for-each (lambda (b) (format #t \"~2,'0x\" b)) (bytevector->u8-list (nix-base32-string->bytevector (cadr (command-line))))) (newline)" \
+  "$expected_nix_base32")
+
+printf 'Using Guile: %s\n' "$guile_bin"
+printf 'Installing into existing prefix: %s\n' "$install_prefix"
+printf 'Working directory: %s\n' "$workdir"
+printf 'Fetching: %s\n' "$source_url"
+fetch -o "$src_tarball" "$source_url"
+actual_sha256_hex=$(sha256 -q "$src_tarball")
+if [ "$actual_sha256_hex" != "$expected_sha256_hex" ]; then
+  echo "sha256 mismatch for $src_tarball" >&2
+  echo "expected: $expected_sha256_hex" >&2
+  echo "actual:   $actual_sha256_hex" >&2
+  exit 1
+fi
+
+bsdtar -xf "$src_tarball" -C "$workdir"
+if [ ! -d "$src_dir" ]; then
+  echo "expected extracted source directory not found: $src_dir" >&2
+  exit 1
+fi
+
+rm -rf "$site_dir/json" "$site_dir/json.scm" "$site_ccache_dir/json" "$site_ccache_dir/json.go"
+mkdir -p "$build_dir"
+(
+  cd "$build_dir"
+  "$src_dir/configure" --prefix="$install_prefix"
+) >"$configure_log" 2>&1
+(
+  cd "$build_dir"
+  "$make_bin" GUILE_AUTO_COMPILE=0 -j"${JOBS:-$(sysctl -n hw.ncpu 2>/dev/null || echo 1)}"
+) >"$build_log" 2>&1
+(
+  cd "$build_dir"
+  "$make_bin" GUILE_AUTO_COMPILE=0 install
+) >"$install_log" 2>&1
+
+export GUILE_LOAD_PATH="$site_dir${GUILE_LOAD_PATH:+:$GUILE_LOAD_PATH}"
+export GUILE_LOAD_COMPILED_PATH="$site_ccache_dir${GUILE_LOAD_COMPILED_PATH:+:$GUILE_LOAD_COMPILED_PATH}"
+json_check=$("$guile_bin" -c '(use-modules (json)) (define-json-mapping <frob> make-frob frob? json->frob (a frob-a) (b frob-b "bee")) (display (if (equal? (json->frob (open-input-string "{ \"a\": 1, \"bee\": 2 }")) (make-frob 1 2)) "ok" "mismatch")) (newline)')
+if [ "$json_check" != "ok" ]; then
+  echo "(json) module validation failed" >&2
+  exit 1
+fi
+
+gnutls_check=$("$guile_bin" -c '(catch #t (lambda () (use-modules (gnutls)) (display "present") (newline)) (lambda _ (display "missing") (newline)))')
+git_check=$("$guile_bin" -c '(catch #t (lambda () (use-modules (git)) (display (if (procedure? graph-descendant?) "present" "missing-export")) (newline)) (lambda _ (display "missing") (newline)))')
+
+cat >"$env_file" <<EOF
+export GUILE_EXTRA_PREFIX='$install_prefix'
+export GUILE_LOAD_PATH='$site_dir'
+export GUILE_LOAD_COMPILED_PATH='$site_ccache_dir'
+export GUILE_EXTENSIONS_PATH='$extensions_dir'
+export LD_LIBRARY_PATH='$install_prefix/lib:$guile_lib_dir:/usr/local/lib'
+EOF
+
+cat >"$metadata_file" <<EOF
+version=$version
+source_url=$source_url
+expected_nix_base32=$expected_nix_base32
+expected_sha256_hex=$expected_sha256_hex
+actual_sha256_hex=$actual_sha256_hex
+guile_bin=$guile_bin
+guile_version=$guile_version
+install_prefix=$install_prefix
+site_dir=$site_dir
+site_ccache_dir=$site_ccache_dir
+extensions_dir=$extensions_dir
+configure_log=$configure_log
+build_log=$build_log
+install_log=$install_log
+json_module_check=$json_check
+gnutls_module_already_present=$gnutls_check
+guile_git_module_already_present=$git_check
+env_file=$env_file
+EOF
+
+if [ -n "${METADATA_OUT:-}" ]; then
+  mkdir -p "$(dirname "$METADATA_OUT")"
+  cp "$metadata_file" "$METADATA_OUT"
+fi
+if [ -n "${ENV_OUT:-}" ]; then
+  mkdir -p "$(dirname "$ENV_OUT")"
+  cp "$env_file" "$ENV_OUT"
+fi
+
+printf 'PASS local-guile-json-build\n'
+printf 'json module check: %s\n' "$json_check"
+printf 'Existing (gnutls) module in prefix: %s\n' "$gnutls_check"
+printf 'Existing (git) module in prefix: %s\n' "$git_check"
+printf 'Environment file: %s\n' "$env_file"
+printf 'Metadata file: %s\n' "$metadata_file"
+if [ -n "${ENV_OUT:-}" ]; then
+  printf 'Copied environment file to: %s\n' "$ENV_OUT"
+fi
+if [ -n "${METADATA_OUT:-}" ]; then
+  printf 'Copied metadata file to: %s\n' "$METADATA_OUT"
+fi
+printf '%s\n' '--- metadata ---'
+cat "$metadata_file"
--- a/tests/guix/run-derivation-generation-investigation.sh
+++ b/tests/guix/run-derivation-generation-investigation.sh
@@ -141,6 +141,17 @@ if [ "$check_rc" -eq 0 ] && [ "$(tr -d '\n' < "$git_check_out")" = ok ]; then
  local_git_check=present
 fi

+json_check_out=$workdir/guile-json-check.out
+json_check_err=$workdir/guile-json-check.err
+local_json_check=missing
+set +e
+"$guile_bin" -c '(use-modules (json)) (define-json-mapping <frob> make-frob frob? json->frob (a frob-a) (b frob-b "bee")) (display (if (equal? (json->frob (open-input-string "{ \"a\": 1, \"bee\": 2 }")) (make-frob 1 2)) "ok" "mismatch")) (newline)' >"$json_check_out" 2>"$json_check_err"
+check_rc=$?
+set -e
+if [ "$check_rc" -eq 0 ] && [ "$(tr -d '\n' < "$json_check_out")" = ok ]; then
+  local_json_check=present
+fi
+
 if [ "$unsupported_rc" -eq 0 ]; then
  echo "configure without --with-courage unexpectedly succeeded" >&2
  exit 1
@@ -170,6 +181,17 @@ if [ "$local_gnutls_check" = present ] && [ "$local_git_check" = missing ]; then
    exit 1
  fi
 fi
+if [ "$local_gnutls_check" = present ] && [ "$local_git_check" = present ] && [ "$local_json_check" = missing ]; then
+  if ! grep -q "Guile-JSON is missing" "$courage_log"; then
+    echo "configure with --with-courage failed, but not for the expected missing-Guile-JSON reason" >&2
+    exit 1
+  fi
+elif [ "$local_json_check" = present ]; then
+  if grep -q "Guile-JSON is missing" "$courage_log"; then
+    echo "configure with --with-courage still reports missing Guile-JSON even though (json) works" >&2
+    exit 1
+  fi
+fi

 cat >"$metadata_file" <<EOF
 source_repo=$source_repo
@@ -188,6 +210,7 @@ configure_with_courage_rc=$courage_rc
 configure_with_courage_error=${courage_error:-<none>}
 local_guile_gnutls_module=$local_gnutls_check
 local_guile_git_module=$local_git_check
+local_guile_json_module=$local_json_check
 unsupported_summary_begin
 $unsupported_summary
 unsupported_summary_end
@@ -208,8 +231,10 @@ if [ "$local_gnutls_check" = missing ]; then
  printf 'Finding 2: configure with --with-courage is currently blocked by missing Guile (gnutls) bindings.\n'
 elif [ "$local_git_check" = missing ]; then
  printf 'Finding 2: Guile (gnutls) is available, but configure is now blocked by missing Guile-Git.\n'
+elif [ "$local_json_check" = missing ]; then
+  printf 'Finding 2: Guile (gnutls) and Guile-Git are available, but configure is now blocked by missing Guile-JSON.\n'
 else
-  printf 'Finding 2: Guile (gnutls) and Guile-Git are available; next blocker: %s\n' "${courage_error:-unknown configure failure}"
+  printf 'Finding 2: Guile (gnutls), Guile-Git, and Guile-JSON are available; next blocker: %s\n' "${courage_error:-unknown configure failure}"
 fi
 if [ "$cleanup" -eq 0 ]; then
  printf 'Metadata file: %s\n' "$metadata_file"