This test had been failing for a few months in part due to this change at
www.gnu.org.
* tests/derivations.scm ("fixed-output derivation, network access, external host"):
Pass #:headers to ‘http-get’.
Change-Id: Ife54e65b2665704af433df669df0d4010b49884c
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Merges: #7470
The 'map-derivation' procedure was trying to process directories as files.
When a derivation had a 'module import' directory as input, it threw an
exception since it tried to open it as a file.
* guix/derivations.scm (map-derivation): In ‘sources’, add
‘file-is-directory?’ case.
* tests/derivations.scm ("map-derivation, modules"): New test.
Fixes: https://issues.guix.gnu.org/71941
Change-Id: I9b766f9aaa03ea9307f73e8abb36bc347af4b5e6
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes a regression introduced in 7173c2c0ca. Additional discussion at
https://codeberg.org/guix/guix/pulls/5977.
* nix/libutil/util.cc (_deletePathAt): chmod directory and retry open when it
fails with EACCES. Do this using an O_PATH file descriptor referenced via
/proc/self/fd whenever possible to avoid it being replaced by a
non-directory immediately before being chmod'ed.
* nix/libutil/util.hh (deletePath): document TOCTTOU race on non-linux systems
where hardlinks aren't protected.
* tests/derivations.scm ("unreadable directories in build tree can be
removed"): new test.
Fixes: guix/guix#5891
Reported-by: Liliana Marie Prikler <liliana.prikler@gmail.com>
Change-Id: I749127fe5254ebabc8387a2f0ef47e3c116bfcc5
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Merges: #6460
While 'url-fetch*' in (guix download) special-cases these URIs, 'git-fetch'
does not. Consequently, the recent changes to (guix scripts perform-download)
that disallow these URIs cause tests that use builtin:git-download to fail.
* guix/tests/git.scm (serve-git-repository, call-with-served-git-repository):
new procedures.
(with-served-git-repository, with-served-temporary-git-repository): new
syntax.
* .dir-locals.el (scheme-mode): add indentation information for
'with-served-git-repository'.
* tests/builders.scm ("git-fetch, file URI"): use git:// URI with
'with-served-temporary-git-repository'.
* tests/derivations.scm ("'git-download' build-in builder, invalid hash",
"'git-download' built-in builder, invalid commit", "'git-download' built-in
builder, not found"): same.
("'git-download' built-in builder"): same, and use a nonce in the repo
contents so that success isn't cached.
Change-Id: Id3e1233bb74d5987faf89c4341e1d37f09c77c80
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Previously, the builder of a fixed-output derivation could communicate with an
external process via an abstract Unix-domain socket. In particular, it could
send an open file descriptor to the store, granting write access to some of
its output files in the store provided the derivation build fails—the fix for
CVE-2024-27297 did not address this specific case. It could also send an open
file descriptor to a setuid program, which could then be executed using
execveat to gain the privileges of the build user.
With this change, fixed-output derivations other than “builtin:download”
and “builtin:git-download” always run in a separate network namespace
and have network access provided by a TAP device backed by slirp4netns,
thereby closing the abstract Unix-domain socket channel.
* nix/libstore/globals.hh (Settings)[useHostLoopback, slirp4netns]: new
fields.
* config-daemon.ac (SLIRP4NETNS): new C preprocessor definition.
* nix/libstore/globals.cc (Settings::Settings): initialize them to defaults.
* nix/nix-daemon/guix-daemon.cc (options): add --isolate-host-loopback option.
* doc/guix.texi: document it.
* nix/libstore/build.cc (DerivationGoal)[slirp]: New field.
(setupTap, setupTapAction, waitForSlirpReadyAction, enableRouteLocalnetAction,
prepareSlirpChrootAction, spawnSlirp4netns, haveGlobalIPv6Address,
remapIdsTo0Action): New functions.
(initializeUserNamespace): allow the guest UID and GID to be specified.
(DerivationGoal::killChild): When ‘slirp’ is not -1, call ‘kill’.
(DerivationGoal::startBuilder): Unconditionally add CLONE_NEWNET to FLAGS.
When ‘fixedOutput’ is true, spawn ‘slirp4netns’.
When ‘fixedOutput’ and ‘useChroot’ are true, add setupTapAction,
waitForSlirpReadyAction, and enableRouteLocalnetAction to builder setup
phases.
Create a /etc/resolv.conf for fixed-output derivations that directs them to
slirp4netns's dns address.
When settings.useHostLoopback is true, supply fixed-output derivations with a
/etc/hosts that resolves "localhost" to slirp4netns's address for accessing
the host loopback.
* nix/libutil/util.cc (keepOnExec, decodeOctalEscaped, sendFD, receiveFD,
findProgram): New functions.
* nix/libutil/util.hh (keepOnExec, decodeOctalEscaped, sendFD, receiveFD,
findProgram): New declarations.
* gnu/packages/package-management.scm (guix): add slirp4netns input for linux
targets.
* tests/derivations.scm (builder-network-isolated?): new variable.
("fixed-output derivation, network access, localhost", "fixed-output
derivation, network access, external host"):
skip test case if fixed output derivations are isolated from the network.
Change-Id: Ia3fea2ab7add56df66800071cf15cdafe7bfab96
Signed-off-by: John Kehayias <john.kehayias@protonmail.com>
These missing inputs go unnoticed when running ‘guix-daemon
--disable-chroot’ but are immediately visible otherwise.
* tests/derivations.scm ("fixed-output derivation"): Add %BASH to #:sources.
("fixed-output derivation: output paths are equal"):
("fixed-output derivation, recursive"):
("derivation with a fixed-output input"):
("derivation with duplicate fixed-output inputs"):
("derivation with equivalent fixed-output inputs"):
("build derivation with coreutils"): Likewise.
* tests/packages.scm (bootstrap-binary): New procedure.
("package-source-derivation, origin, sha512"): Use it instead of
‘search-bootstrap-binary’ and add BASH to #:sources.
("package-source-derivation, origin, sha3-512"): Likewise.
Change-Id: I4c9087df23c47729a3aff15e9e1435b7266e36e2
Many thanks to Reepca Russelstein for their review and guidance on these
changes.
* nix/libstore/build.cc (guestUID, guestGID): New variables.
(DerivationGoal)[readiness]: New field.
(initializeUserNamespace): New function.
(DerivationGoal::runChild): When ‘readiness.readSide’ is positive, read
from it.
(DerivationGoal::startBuilder): Call ‘chown’
only when ‘buildUser.enabled()’ is true. Pass CLONE_NEWUSER to ‘clone’
when ‘buildUser.enabled()’ is false or not running as root. Retry
‘clone’ without CLONE_NEWUSER upon EPERM.
(DerivationGoal::registerOutputs): Make ‘actualPath’ writable before
‘rename’.
(DerivationGoal::deleteTmpDir): Catch ‘SysError’ around ‘_chown’ call.
* nix/libstore/local-store.cc (LocalStore::createUser): Do nothing if
‘dirs’ already exists. Warn instead of failing when failing to chown
‘dir’.
* guix/substitutes.scm (%narinfo-cache-directory): Check for
‘_NIX_OPTIONS’ rather than getuid() == 0 to determine the cache
location.
* doc/guix.texi (Build Environment Setup): Reorganize a bit. Add
section headings “Daemon Running as Root” and “The Isolated Build
Environment”. Add “Daemon Running Without Privileges” subsection.
Remove paragraph about ‘--disable-chroot’.
(Invoking guix-daemon): Warn against ‘--disable-chroot’ and explain why.
* tests/derivations.scm ("builder is outside the store"): New test.
Reviewed-by: Reepca Russelstein <reepca@russelstein.xyz>
That makes ‘derivation-build-plan’ directly usable in cases where one
wants to sequentially build derivations one by one, or to report builds
in the right order in the user interface.
* guix/derivations.scm (derivation-build-plan): Wrap ‘loop’ in
‘traverse’. Perform a depth-first traversal. Return the list of builds
in topological order.
* tests/derivations.scm ("derivation-build-plan, topological ordering"):
New test.
Change-Id: I7cd9083f42c4381b4213794a40dbb5b234df966d
The new builder makes it possible to break cycles that occurs when the
fixed-output derivation for the source of a dependency of ‘git’ would
itself depend on ‘git’.
* guix/scripts/perform-download.scm (perform-git-download): New
procedure.
(perform-download): Move fixed-output derivation check to…
(guix-perform-download): … here. Invoke ‘perform-download’ or
‘perform-git-download’ depending on what ‘derivation-builder’ returns.
* nix/libstore/builtins.cc (builtins): Add “git-download”.
* tests/derivations.scm ("built-in-builders"): Update.
("'git-download' built-in builder")
("'git-download' built-in builder, invalid hash")
("'git-download' built-in builder, invalid commit")
("'git-download' built-in builder, not found"): New tests.
* nix/libutil/hash.cc (parseHash): Show the hash algorithm name and
expected size in the error message.
* tests/derivations.scm ("fixed-output derivation, invalid hash size"):
New test.
Reported by Stephen Paul Weber <singpolyma@singpolyma.net> at
<https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00035.html>.
* guix/derivations.scm (read-derivation)[outputs->alist]: Treat the
empty hash case as non-fixed-output whether or not the hash algorithm is
the empty string, and preserve the hash algorithm in <derivation-output>.
* tests/derivations.scm ("'download' built-in builder, no fixed-output hash")
("fixed-output-derivation?, no hash", "read-derivation with hash = #f"): New tests.
These modules would use (guix grafts) just to access '%graft?' and
related bindings, which are now in (guix store).
* gnu/ci.scm,
guix/gexp.scm,
guix/lint.scm,
guix/scripts.scm,
guix/scripts/archive.scm,
guix/scripts/build.scm,
guix/scripts/challenge.scm,
guix/scripts/deploy.scm,
guix/scripts/environment.scm,
guix/scripts/home.scm,
guix/scripts/pack.scm,
guix/scripts/package.scm,
guix/scripts/pull.scm,
guix/scripts/size.scm,
guix/scripts/system.scm,
guix/scripts/weather.scm,
tests/builders.scm,
tests/channels.scm,
tests/cpan.scm,
tests/derivations.scm,
tests/gexp.scm,
tests/graph.scm,
tests/guix-daemon.sh,
tests/monads.scm,
tests/pack.scm,
tests/packages.scm,
tests/profiles.scm,
tests/system.scm: Remove #:use-module (guix grafts).
Fixes <https://issues.guix.gnu.org/54209>.
* guix/derivations.scm (coalesce-duplicate-inputs): Use the output paths
of DRV as a hash table key.
* tests/derivations.scm ("derivation with duplicate fixed-output
inputs"): Expect a single input for FINAL.
("derivation with equivalent fixed-output inputs"): New test.
Files smaller than 8 KiB typically represent ~70% of the entries in
/gnu/store/.links but only contribute to ~4% of the space savings
afforded by deduplication.
Not considering these files for deduplication speeds up file insertion
in the store and, more importantly, leaves 'removeUnusedLinks' with
fewer entries to traverse, thereby speeding it up proportionally.
Partly fixes <https://issues.guix.gnu.org/24937>.
* config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK
definition.
* guix/store/deduplication.scm (%deduplication-minimum-size): New
variable.
(deduplicate)[loop]: Do not recurse when FILE's size is below
%DEDUPLICATION-MINIMUM-SIZE.
(dump-port): New procedure.
(dump-file/deduplicate)[hash]: Turn into...
[dump-and-compute-hash]: ... this thunk.
Call 'deduplicate' only when SIZE is greater than
%DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where
st.st_size < deduplicationMinSize.
* nix/libstore/local-store.hh (deduplicationMinSize): New declaration.
* nix/libstore/optimise-store.cc (deduplicationMinSize): New variable.
(LocalStore::optimisePath_): Return when PATH is a symlink or smaller
than 'deduplicationMinSize'.
* tests/derivations.scm ("identical files are deduplicated"): Produce
files bigger than %DEDUPLICATION-MINIMUM-SIZE.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
Likewise.
* tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"):
New test.
("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than
%DEDUPLICATION-MINIMUM-SIZE.
* tests/store.scm ("substitute, deduplication"): Likewise.
Previously, test cases could fail if some process was listening
at a hard-coded port. This patch eliminates most of these potential
failures, by automatically assigning an unbound port. This should
allow for building multiple guix trees in parallel outside a build
container, though this is currently untested.
The test "home-page: Connection refused" in tests/lint.scm still
hardcodes port 9999, however.
* guix/tests/http.scm
(http-server-can-listen?): remove now unused procedure.
(%http-server-port): default to port 0, meaning the OS
will automatically choose a port.
(open-http-server-socket): remove the false statement claiming
this procedure is exported and also return the allocated port
number.
(%local-url): raise an error if the port is obviously unbound.
(call-with-http-server): set %http-server-port to the allocated
port while the thunk is called.
* tests/derivations.scm: adjust test cases to use automatically
assign a port. As there is no risk of a port conflict now,
do not make any tests conditional upon 'http-server-can-listen?'
anymore.
* tests/elpa.scm: likewise.
* tests/lint.scm: likewise, and add a TODO comment about a port
that is still hard-coded.
* tests/texlive.scm: likewise.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
* guix/derivations.scm (derivation-input-fold): New procedure.
(substitution-oracle)[closure]: Rewrite in terms of
'derivation-input-fold'.
* tests/derivations.scm ("derivation-input-fold"): New test.
* guix/tests/http.scm (call-with-http-server): Replace 'code' and 'data'
parameters with 'responses+data'. Compute RESPONSES as a function of
that. Remove #:headers parameter.
[http-write]: Quit only when RESPONSES is empty.
[server-body]: Get the response and data from RESPONSES, and set it to
point to the rest.
(with-http-server): Adjust accordingly.
* tests/derivations.scm ("'download' built-in builder")
("'download' built-in builder, invalid hash")
("'download' built-in builder, not found")
("'download' built-in builder, check mode"): Adjust to new
'with-http-server' interface.
* tests/lint.scm ("home-page: 200")
("home-page: 200 but short length")
("home-page: 404", "home-page: 301, invalid"):
("home-page: 301 -> 200", "home-page: 301 -> 404")
("source: 200", "source: 200 but short length")
("source: 404", "source: 404 and 200")
("source: 301 -> 200", "source: 301 -> 404"):
("github-url", github-url): Likewise.
* tests/swh.scm (with-json-result)
("lookup-origin, not found"): Likewise.
Fixes <https://bugs.gnu.org/36777>.
Reported by Carl Dong <contact@carldong.me>.
* guix/derivations.scm (derivation/masked-inputs): Call
'delete-duplicates' on INPUTS.
* tests/derivations.scm ("derivation with duplicate fixed-output inputs"):
New test.
This fixes a bug whereby "guix build texlive -n" would report:
0.0 MB would be downloaded:
/gnu/store/…-texlive-20180414
instead of:
The following derivation would be built:
/gnu/store/…-texlive-texmf-20180414.drv
2,595.2 MB would be downloaded:
/gnu/store/…-texlive-20180414-texmf.tar.xz
/gnu/store/…-texlive-20180414
where 'texlive-texmf' is a non-substitutable dependency of 'texlive'.
* guix/derivations.scm (dependencies-of-substitutables): New procedure.
(derivation-build-plan): When 'input-substitutable-info' returns true,
append the subset of DEPS that corresponds to SUBSTITUABLES to the first
argument of 'loop'.
* guix/ui.scm (show-what-to-build): Remove half-baked traversal of
DOWNLOAD.
* tests/derivations.scm ("derivation-build-plan and substitutes,
non-substitutable dep"): New test.
* guix/derivations.scm (build-derivations): Accept <derivation-input>
records among DERIVATIONS.
* tests/derivations.scm ("build-derivations with specific output"): Test
it.
Consequently, the whole graph of <derivation> object is readily
available without having to go through 'read-derivation-from-file',
which could have cache misses if the requested <derivation> object had
been GC'd in the meantime. This is an important property for the
performance of things like 'derivation-build-plan' that traverse the
derivation graph.
* guix/derivations.scm (<derivation-input>): Replace 'path' field by
'derivation'.
(derivation-input-path): Adjust accordingly.
(derivation-input-key): New procedure.
(derivation-input-output-paths): Adjust accordingly.
(coalesce-duplicate-inputs): Likewise.
(derivation-prerequisites): Use 'derivation-input-key' to compute keys
for INPUT-SET.
(derivation-build-plan): Likewise.
(read-derivation): Add optional 'read-derivation-from-file' parameter.
[make-input-drvs]: Call it.
(write-derivation)[write-input]: Adjust to new <derivation-input>.
(derivation/masked-inputs): Likewise, and remove redundant
'coalesce-duplicate-inputs' call.
(derivation)[input->derivation-input]: Change to consider only the
derivation case. Update call to 'make-derivation-input'.
[input->source]: New procedure.
Separate sources from inputs.
(map-derivation): Adjust to new <derivation-input>.
* tests/derivations.scm ("parse & export"): Pass a second argument to
'read-derivation'.
("build-expression->derivation and derivation-prerequisites")
("derivation-prerequisites and valid-derivation-input?"): Adjust to new
<derivation-input>.
The new 'derivation-build-plan' procedure has a more appropriate
signature: it takes a list of <derivation-inputs> instead of taking one
<derivation>. Its body is also much simpler.
* guix/derivations.scm (derivation-build-plan): New procedure.
(derivation-prerequisites-to-build): Express in terms of
'derivation-build-plan' and mark as deprecated.
* tests/derivations.scm: Change 'derivation-prerequisites-to-build'
tests to 'derivation-build-plan' and adjust accordingly.
* tests/derivations.scm ("derivation-prerequisites-to-build when outputs
already present"): Insert random text in BUILDER, and remove two calls
to 'delete-paths'. Use 'derivation->output-path' for INPUT-PATH instead
of an old idiom.
This allows callers to request the substitution of a single derivation
output.
* guix/store.scm (build-things): Accept derivation/output pairs among
THINGS.
* guix/derivations.scm (build-derivations): Likewise.
* tests/store.scm ("substitute + build-things with specific output"):
New test.
* tests/derivations.scm ("build-derivations with specific output"):
New test.
* doc/guix.texi (The Store): Adjust accordingly.
* guix/derivations.scm (derivation-prerequisites-to-build): Rename
#:substitutable? to #:substitutable-info.
[derivation-substitutable?]: Rename to...
[derivation-substitutable-info]: ... this. Return a list of <substitutable>.
Second return value is now a list of <substitutable> instead of a list
of strings.
* guix/ui.scm (show-what-to-build)[substitutable?]: Rename to...
[substitutable-info]: ... this.
Adjust to new 'derivation-prerequisites-to-build' return value type.
* tests/derivations.scm ("derivation-prerequisites-to-build and
substitutes"): Adjust.
("derivation-prerequisites-to-build and substitutes, local build"):
Likewise.
* guix/derivations.scm (substitution-oracle): Use
'substitution-path-info' instead of 'substitution-paths'. Turn SUBST
into a vhash from path to <substitutable>. Change the returned
procedure to provide a <substitutable> instead of a Boolean.
* tests/derivations.scm ("substitution-oracle and #:substitute? #f"):
Mock 'substitutable-path-info' instead of 'substitutable-paths'.
This is a followup to 9231ef12f2.
* tests/derivations.scm ("build-expression->derivation and builder
encoding"): Set '%default-port-encoding' to "UTF-8".
This avoids connections to substitute servers for derivations that are
not substitutable anyway, such as profiles.
Reported by Andy Wingo.
* guix/derivations.scm (substitution-oracle): Skip derivations that do
not pass 'substitutable-derivation?'.
* tests/derivations.scm ("substitution-oracle and #:substitute? #f"):
New test.
