Fixes a regression on Guix System introduced in
c4298638ca (from guix/guix#4178) where files in
/etc, such as /etc/services, would be symlinks. Consequently, the chroot of
fixed-output derivations would contain a dangling symlink for /etc/services,
leading to name resolution failures in the chroot.
This also relates to 82f84f5e7f.
* nix/libstore/build.cc (DerivationGoal::startBuilder): Add call to
‘canonPath’.
Change-Id: If34c8e74447ffc03d9fa81a1ea6bc0aef7c4d0b3
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
* nix/libstore/build.cc (prepareSlirpChrootAction): Use
canonPath(i, true) to resolve symlinks when adding /etc/resolv.conf
and /etc/hosts to the slirp4netns chroot, so that bindMount receives
a regular file path instead of a symlink.
On systems using systemd-resolved, /etc/resolv.conf is typically a
symlink:
/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
The slirp4netns chroot creates an empty /run/ directory, so when
bindMount copies the symlink verbatim (spawn.cc line 537-542), the
target does not exist and slirp4netns cannot determine the upstream
DNS server. This causes all DNS resolution to fail for fixed-output
derivations that use the Guile-based git-fetch builder (e.g.
git-fetch/lfs), since they rely on slirp4netns for network access
in the build chroot.
Derivations using builtin:git-download are unaffected because they
run in the daemon process itself, which has full network access.
Change-Id: Ib73e69a8760e74eb8141dd0408c27aa8b3001e37
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Merges: #6959
Fixes <https://issues.guix.gnu.org/77862>.
Previously, the ‘guix-daemon’ account (for unprivileged execution) would
typically have “kvm” as a supplementary group, but that group would not
be mapped in the build user namespace. Consequently, attempts to
‘chown’ a file to that supplementary group would fail with EINVAL.
The test suites of Coreutils, Python, and Go (among others) exercise
this chown-to-supplementary-group behavior, so they would all fail when
started by the unprivileged ‘guix-daemon’ even though they succeed when
started by ‘guix-daemon’ running as root.
Thanks to keinflue <keinflue@posteo.net> and Reepca Russelstein
<reepca@russelstein.xyz> for helping out.
* nix/libstore/build.cc (initializeUserNamespace): Add ‘extraGIDs’ and
‘haveCapSetGID’ parameters. Invoke ‘newgidmap’ when ‘extraGIDs’ is
non-empty and ‘haveCapSetGID’ is false. Honor ‘extraGIDs’ when
‘haveCapSetGID’ is true.
(maxGroups, guestKVMGID): New variables.
(kvmGIDMapping): New function.
(DerivationGoal::startBuilder): Set ‘ctx.lockMountsMapAll’ in the
CLONE_NEWUSER case. Pass ‘extraGIDs’ to ‘initializeUserNamespace’.
* tests/store.scm ("kvm GID is mapped"): New test.
Change-Id: I10ba710fc1b9ca1e3cd3122be1ec8ede5df18b40
At this point the value of 'st.st_mode' is from before canonicalization, so
restoring to that will undo the permissions aspect of the canonicalization for
a top-level directory store item.
Fixes#1104, introduced in
ae18b3d9e6 (ae18b3d9e6).
* nix/libstore/build.cc (DerivationGoal::registerOutputs): update 'st' with
post-canonicalization permissions before making actualPath
temporarily-writable.
* tests/store.scm ("build outputs aren't writable"): new test.
Change-Id: I5e5eaa79fa6b7f81e1d12fd285883c762a22ce5a
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://issues.guix.gnu.org/78919>.
Fixesguix/guix#471.
Fixes a bug introduced in ae18b3d9e6
whereby interrupting ‘guix build -K hello’ would leave a build directory
with root ownership due ‘secureFilePerms’ bailing out due to the
presence of FIFOs in the temporary build directory.
* nix/libstore/build.cc (secureFilePerms): Add ‘allowSpecialFiles’
parameter; honor it and pass it in recursive call.
(DerivationGoal::deleteTmpDir): Pass true as the second argument to
‘secureFilePerms’.
Reported-by: Janneke Nieuwenhuizen <janneke@gnu.org>
Reported-by: David Elsing <david.elsing@posteo.net>
Change-Id: I638a4ee909a2b5022f9153e1cbb832bfb2e15263
glibc currently will insist on using 'socketcall' on i686-linux unless built
with '--enable-kernel=4.3.0' or above, even on systems that have dedicated
system calls available for all the socket-related functionality. This
behavior breaks the assumption that socketcall can be safely blocked without
impacting functionality in slirp4netns, rendering the seccomp filter unusable
with those glibcs.
This change makes the slirp4netns seccomp filter opt-in on systems with a
'socketcall' system call. It can either be opted-into at compile-time or at
runtime using the NO_SOCKETCALL_LIBC preprocessor define or the
GUIX_FORCE_SECCOMP environment variable, respectively.
The seccomp filter being disabled on these systems means that it is possible
for a compromised slirp4netns to access abstract unix domain sockets in the
root network namespace. It does not affect any of the other mechanisms used
to isolate slirp4netns (e.g. chroot, namespaces, etc).
Fixesguix/guix#808.
* nix/libstore/build.cc (spawnSlirp4netns) [__NR_socketcall]: Do not add
seccomp filter, unless ‘GUIX_FORCE_SECCOMP’ is set.
Change-Id: Ibfe8becc9431f5aff11a21f06858b20496f9cb4a
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
The container that slirp4netns runs in should already be quite difficult to do
anything malicious in beyond basic denial of service or sending of network
traffic. There is, however, one hole remaining in the case in which there is
an adversary able to run code locally: abstract unix sockets. Because these
are governed by network namespaces, not IPC namespaces, and slirp4netns is in
the root network namespace, any process in the root network namespace can
cooperate with the slirp4netns process to take over its user.
To close this, we use seccomp to block the creation of unix-domain sockets by
slirp4netns. This requires some finesse, since slirp4netns absolutely needs
to be able to create other types of sockets - at minimum AF_INET and AF_INET6
Seccomp has many, many pitfalls. To name a few:
1. Seccomp provides you with an "arch" field, but this does not uniquely
determine the ABI being used; the actual meaning of a system call number
depends on both the number (which is often the result of ORing a related
system call with a flag for an alternate ABI) and the architecture.
2. Seccomp provides no direct way of knowing what the native value for the
arch field should be; the user must do configure/compile-time testing for
every architecture+ABI combination they want to support. Amusingly enough,
the linux-internal header files have this exact information
(SECCOMP_ARCH_NATIVE), but they aren't sharing it.
3. The only system call numbers we naturally have are the native ones in
asm/unistd.h. __NR_socket will always refer to the system call number for
the target system's ABI.
4. Seccomp can only manipulate 32-bit words, but represents every system call
argument as a uint64.
5. New system call numbers with as-yet-unknown semantics can be added to the
kernel at any time.
6. Based on this comment in arch/x86/entry/syscalls/syscall_32.tbl:
# 251 is available for reuse (was briefly sys_set_zone_reclaim)
previously-invalid system call numbers may later be reused for new system
calls.
7. Most architecture+ABI combinations have system call tables with many gaps
in them. arm-eabi, for example, has 35 such gaps (note: this is just the
number of distinct gaps, not the number of system call numbers contained in
those gaps).
8. Seccomp's BPF filters require a fully-acyclic control flow graph.
Any operation on a data structure must therefore first be fully
unrolled before it can be run.
9. Seccomp cannot dereference pointers. Only the raw bits provided to the
system calls can be inspected.
10. Some architecture+ABI combos have multiplexer system calls. For example,
socketcall can perform any socket-related system call. The arguments to
the multiplexed system call are passed indirectly, via a pointer to user
memory. They therefore cannot be inspected by seccomp.
11. Some valid system calls are not listed in any table in the kernel source.
For example, __ARM_NR_cacheflush is an "ARM private" system call. It does
not appear in any *.tbl file.
12. Conditional branches are limited to relative jumps of at most 256
instructions forward.
13. Prior to Linux 4.8, any process able to spawn another process and call
ptrace could bypass seccomp restrictions.
To address (1), (2), and (3), we include preprocessor checks to identify the
native architecture value, and reject all system calls that don't use the
native architecture.
To address (4), we use the AC_C_BIGENDIAN autoconf check to conditionally
define WORDS_BIGENDIAN, and match up the proper portions of any uint64 we test
for with the value in the accumulator being tested against.
To address (5) and (6), we use system call pinning. That is, we hardcode a
snapshot of all the valid system call numbers at the time of writing, and
reject any system call numbers not in the recorded set. A set is recorded for
every architecture+ABI combo, and the native one is chosen at compile-time.
This ensures that not only are non-native architectures rejected, but so are
non-native ABIs. For the sake of conciseness, we represent these sets as sets
of disjoint ranges. Due to (7), checking each range in turn could add a lot
of overhead to each system call, so we instead binary search through the
ranges. Due to (8), this binary search has to be fully unrolled, so we do
that too.
It can be tedious and error-prone to manually produce the syscall ranges by
looking at linux's *.tbl files, since the gaps are often small and
uncommented. To address this, a script, build-aux/extract-syscall-ranges.sh,
is added that will produce them given a *.tbl filename and an ABI regex (some
tables seem to abuse the ABI field with strange values like "memfd_secret").
Note that producing the final values still requires looking at the proper
asm/unistd.h file to find any private numbers and to identify any offsets and
ABI variants used.
(10) used to have no good solution, but in the past decade most architectures
have gained dedicated system call alternatives to at least socketcall, so we
can (hopefully) just block it entirely.
To address (13), we block ptrace also.
* build-aux/extract-syscall-ranges.sh: new script.
* Makefile.am (EXTRA_DIST): register it.
* config-daemon.ac: use AC_C_BIGENDIAN.
* nix/libutil/spawn.cc (setNoNewPrivsAction, addSeccompFilterAction): new
functions.
* nix/libutil/spawn.hh (setNoNewPrivsAction, addSeccompFilterAction): new
declarations.
(SpawnContext)[setNoNewPrivs, addSeccompFilter]: new fields.
* nix/libutil/seccomp.hh: new header file.
* nix/libutil/seccomp.cc: new file.
* nix/local.mk (libutil_a_SOURCES, libutil_headers): register them.
* nix/libstore/build.cc (slirpSeccompFilter, writeSeccompFilterDot):
new functions.
(spawnSlirp4netns): use them, set seccomp filter for slirp4netns.
Change-Id: Ic92c7f564ab12596b87ed0801b22f88fbb543b95
Signed-off-by: John Kehayias <john.kehayias@protonmail.com>
Previously, the builder of a fixed-output derivation could communicate with an
external process via an abstract Unix-domain socket. In particular, it could
send an open file descriptor to the store, granting write access to some of
its output files in the store provided the derivation build fails—the fix for
CVE-2024-27297 did not address this specific case. It could also send an open
file descriptor to a setuid program, which could then be executed using
execveat to gain the privileges of the build user.
With this change, fixed-output derivations other than “builtin:download”
and “builtin:git-download” always run in a separate network namespace
and have network access provided by a TAP device backed by slirp4netns,
thereby closing the abstract Unix-domain socket channel.
* nix/libstore/globals.hh (Settings)[useHostLoopback, slirp4netns]: new
fields.
* config-daemon.ac (SLIRP4NETNS): new C preprocessor definition.
* nix/libstore/globals.cc (Settings::Settings): initialize them to defaults.
* nix/nix-daemon/guix-daemon.cc (options): add --isolate-host-loopback option.
* doc/guix.texi: document it.
* nix/libstore/build.cc (DerivationGoal)[slirp]: New field.
(setupTap, setupTapAction, waitForSlirpReadyAction, enableRouteLocalnetAction,
prepareSlirpChrootAction, spawnSlirp4netns, haveGlobalIPv6Address,
remapIdsTo0Action): New functions.
(initializeUserNamespace): allow the guest UID and GID to be specified.
(DerivationGoal::killChild): When ‘slirp’ is not -1, call ‘kill’.
(DerivationGoal::startBuilder): Unconditionally add CLONE_NEWNET to FLAGS.
When ‘fixedOutput’ is true, spawn ‘slirp4netns’.
When ‘fixedOutput’ and ‘useChroot’ are true, add setupTapAction,
waitForSlirpReadyAction, and enableRouteLocalnetAction to builder setup
phases.
Create a /etc/resolv.conf for fixed-output derivations that directs them to
slirp4netns's dns address.
When settings.useHostLoopback is true, supply fixed-output derivations with a
/etc/hosts that resolves "localhost" to slirp4netns's address for accessing
the host loopback.
* nix/libutil/util.cc (keepOnExec, decodeOctalEscaped, sendFD, receiveFD,
findProgram): New functions.
* nix/libutil/util.hh (keepOnExec, decodeOctalEscaped, sendFD, receiveFD,
findProgram): New declarations.
* gnu/packages/package-management.scm (guix): add slirp4netns input for linux
targets.
* tests/derivations.scm (builder-network-isolated?): new variable.
("fixed-output derivation, network access, localhost", "fixed-output
derivation, network access, external host"):
skip test case if fixed output derivations are isolated from the network.
Change-Id: Ia3fea2ab7add56df66800071cf15cdafe7bfab96
Signed-off-by: John Kehayias <john.kehayias@protonmail.com>
This adds a mechanism for manipulating and running "spawn phases" similarly to
how builder-side code manipulates "build phases". The main difference is that
spawn phases take a (reference to a) single structure that they can both read
from and write to, with their writes being visible to subsequent phases. The
base structure type for this is SpawnContext.
It also adds some predefined phase sequences, namely basicSpawnPhases and
cloneSpawnPhases, and exposes each of the actions performed by these phases.
Finally, it modifies build.cc to replace runChild() with use of this new code.
* nix/libutil/util.cc (keepOnExec, waitForMessage): new functions.
* nix/libutil.util.hh (keepOnExec, waitForMessage): add prototypes.
* nix/libutil/spawn.cc, nix/libutil/spawn.hh: new files.
(addPhaseAfter, addPhaseBefore, prependPhase, appendPhase, deletePhase,
replacePhase, reset_writeToStderrAction, restoreAffinityAction,
setsidAction, earlyIOSetupAction, dropAmbientCapabilitiesAction,
chrootAction, chdirAction, closeMostFDsAction, setPersonalityAction,
oomSacrificeAction, setIDsAction, restoreSIGPIPEAction, setupSuccessAction,
execAction, getBasicSpawnPhases, usernsInitSyncAction, usernsSetIDsAction,
initLoopbackAction, setHostAndDomainAction, makeFilesystemsPrivateAction,
makeChrootSeparateFilesystemAction, statfsToMountFlags, bindMount,
mountIntoChroot, mountIntoChrootAction, mountProcAction, mountDevshmAction,
mountDevptsAction, pivotRootAction, lockMountsAction, getCloneSpawnPhases,
runChildSetup, runChildSetupEntry, cloneChild, idMapToIdentityMap,
unshareAndInitUserns): new procedures.
* nix/local.mk (libutil_a_SOURCES): add spawn.cc.
(libutil_headers): add spawn.hh.
* nix/libstore/build.cc (restoreSIGPIPE, DerivationGoal::runChild,
childEntry): removed procedures.
(DerivationGoal::{dirsInChroot,env,readiness}): removed.
(execBuilderOrBuiltin, execBuilderOrBuiltinAction,
clearRootWritePermsAction): new procedures.
(DerivationGoal::startBuilder): modified to use a CloneSpawnContext if
chroot builds are available, otherwise a SpawnContext.
Change-Id: Ifd50110de077378ee151502eda62b99973d083bf
Change-Id: I76e10d3f928cc30566e1e6ca79077196972349f8
spawn.cc, util.cc, util.hh changes
Change-Id: I287320e63197cb4f65665ee5b3fdb3a0e125ebac
Signed-off-by: John Kehayias <john.kehayias@protonmail.com>
Partly fixes <https://issues.guix.gnu.org/77862>.
Fixes a bug whereby, when running guix-daemon unprivileged, /etc/group
would contain the wrong GID for the “nixbld” group. This inconsistency
would lead to failures in the Coreutils test suite, for instance.
* nix/libstore/build.cc (DerivationGoal::startBuilder): Use ‘guestGID’
when writing /etc/group.
* tests/store.scm ("/etc/passwd and /etc/group"): New test.
Reported-by: keinflue <keinflue@posteo.net>
Change-Id: I739bc96c4c935fd9015a45e2bfe5b3e3f90554a9
Fixes <https://issues.guix.gnu.org/77570>.
Commit 40f69b586a made chroot root
directory read-only; as a consequence, build processes attempting to
write to the root directory would now get EROFS instead of EACCES.
It turns out that a number of test suites (Go, Ruby, SCons, Shepherd)
would fail because of this observable difference.
To restore previous behavior in build environments while still
preventing build processes from exposing their root directory to outside
processes, this patch (1) keeps the root writable but #o555 by default,
thereby restoring the EACCES behavior, and (2) ensures that the parent
of the chroot root directory is itself user-accessible only.
* nix/libstore/build.cc (class DerivationGoal)[chrootRootTop]: New
field.
(DerivationGoal::startBuilder): Initialize ‘chrootRootTop’ and make it
‘AutoDelete’. Replace ‘mount’ call that made the root directory
read-only by a mere ‘chmod_’ call.
* tests/store.scm ("build root cannot be made world-readable"): Remove.
("writing to build root leads to EACCES"): New test.
Reported-by: Ada Stevenson <adanskana@gmail.com>
Reported-by: keinflue <keinflue@posteo.net>
Suggested-by: Reepca Russelstein <reepca@russelstein.xyz>
Change-Id: I5912e8b3b293f8242a010cfc79255fc981314445
* config-daemon.ac: Check for <sys/prctl.h>.
* nix/libstore/build.cc (DerivationGoal::runChild): When ‘useChroot’ is
true, call ‘prctl’ to drop all ambient capabilities.
Change-Id: If34637fc508e5fb6d278167f5df7802fc595284f
Many thanks to Reepca Russelstein for their review and guidance on these
changes.
* nix/libstore/build.cc (guestUID, guestGID): New variables.
(DerivationGoal)[readiness]: New field.
(initializeUserNamespace): New function.
(DerivationGoal::runChild): When ‘readiness.readSide’ is positive, read
from it.
(DerivationGoal::startBuilder): Call ‘chown’
only when ‘buildUser.enabled()’ is true. Pass CLONE_NEWUSER to ‘clone’
when ‘buildUser.enabled()’ is false or not running as root. Retry
‘clone’ without CLONE_NEWUSER upon EPERM.
(DerivationGoal::registerOutputs): Make ‘actualPath’ writable before
‘rename’.
(DerivationGoal::deleteTmpDir): Catch ‘SysError’ around ‘_chown’ call.
* nix/libstore/local-store.cc (LocalStore::createUser): Do nothing if
‘dirs’ already exists. Warn instead of failing when failing to chown
‘dir’.
* guix/substitutes.scm (%narinfo-cache-directory): Check for
‘_NIX_OPTIONS’ rather than getuid() == 0 to determine the cache
location.
* doc/guix.texi (Build Environment Setup): Reorganize a bit. Add
section headings “Daemon Running as Root” and “The Isolated Build
Environment”. Add “Daemon Running Without Privileges” subsection.
Remove paragraph about ‘--disable-chroot’.
(Invoking guix-daemon): Warn against ‘--disable-chroot’ and explain why.
* tests/derivations.scm ("builder is outside the store"): New test.
Reviewed-by: Reepca Russelstein <reepca@russelstein.xyz>
* nix/libstore/build.cc (DerivationGoal::runChild): Bind-mount the store
and /tmp under ‘chrootRootDir’ to themselves as read-write.
Remount / as read-only.
Change-Id: I79565094c8ec8448401897c720aad75304fd1948
Those files may be missing in some contexts, for instance within the
build environment.
* nix/libstore/build.cc (DerivationGoal::runChild): Add /etc/resolv.conf
and related files to ‘ss’ only if they exist.
Change-Id: Ie19664a86c8101a1dc82cf39ad4b7abb10f8250a
Fixes <https://issues.guix.gnu.org/31785>.
Similar to <https://github.com/NixOS/nix/issues/178>, fixed in
<29cde917fe>.
We can't rely on Goal deletion to release our locks in a timely manner. In
the case in which multiple guix-daemon processes simultaneously try producing
an output path path1, the one that gets there first (P1) will get the lock,
and the second one (P2) will continue trying to acquire the lock until it is
released. Once it has acquired the lock, it checks to see whether the path
has already become valid in the meantime, and if so it reports success to
those Goals waiting on its completion and finishes. Unfortunately, it fails
to release the locks it holds first, so those stay held until that Goal gets
deleted.
Suppose we have the following store path dependency graph:
path4
/ | \
path1 path2 path3
P2 is now sitting on path1's lock, and will continue to do so until path4 is
completed. Suppose there is also a P3, and it has been blocked while P1
builds path2. Now P3 is sitting on path2's lock, and can't acquire path1's
lock to determine that it has been completed. Likewise P2 is sitting on
path1's lock, and now can't acquire path2's lock to determine that it has been
completed. Finally, P3 completes path3 while P2 is blocked.
Now:
- P1 knows that path1 and path2 are complete, and holds no locks, but can't
determine that path3 is complete
- P2 knows that path1 and path3 are complete, and holds locks on path1 and
path3, but can't determine that path2 is complete
- P3 knows that path2 and path3 are complete, and holds a lock on path2, but
can't determine that path1 is complete
And none of these locks will be released until path4 is complete. Thus, we
have a deadlock.
To resolve this, we should explicitly release these locks as soon as they
should be released.
* nix/libstore/build.cc
(DerivationGoal::tryToBuild, SubstitutionGoal::tryToRun):
Explicitly release locks in the has-become-valid case.
* tests/store-deadlock.scm: New file.
* Makefile.am (SCM_TESTS): Add it.
Change-Id: Ie510f84828892315fe6776c830db33d0f70bcef8
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
There is currently a window of time between when the build outputs are exposed
and when their metadata is canonicalized.
* nix/libstore/build.cc (DerivationGoal::registerOutputs): wait until after
metadata canonicalization to move successful build outputs to the store.
Change-Id: Ia995136f3f965eaf7b0e1d92af964b816f3fb276
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
The only thing keeping a rogue builder and a local user from collaborating to
usurp control over the builder's user during the build is the fact that
whatever files the builder may produce are not accessible to any other users
yet. If we're going to make them accessible, we should probably do some
sanity checking to ensure that sort of collaborating can't happen.
Currently this isn't happening when failed build outputs are moved from the
chroot as an aid to debugging.
* nix/libstore/build.cc (secureFilePerms): new function.
(DerivationGoal::buildDone): use it.
Change-Id: I9dce1e3d8813b31cabd87a0e3219bf9830d8be96
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
This is a followup to 8f4ffb3fae.
Commit 8f4ffb3fae fell short in two
ways: (1) it didn’t have any effet for fixed-output derivations
performed in a chroot, which is the case for all of them except those
using “builtin:download” and “builtin:git-download”, and (2) it did not
preserve ownership when copying, leading to “suspicious ownership or
permission […] rejecting this build output” errors.
* nix/libstore/build.cc (DerivationGoal::buildDone): Account for
‘chrootRootDir’ when copying ‘drv.outputs’.
* nix/libutil/util.cc (copyFileRecursively): Add ‘fchown’ and ‘fchownat’
calls to preserve file ownership; this is necessary for chrooted
fixed-output derivation builds.
* nix/libutil/util.hh: Update comment.
Change-Id: Ib59f040e98fed59d1af81d724b874b592cbef156
This fixes a security issue (CVE-2024-27297) whereby a fixed-output
derivation build process could open a writable file descriptor to its
output, send it to some outside process for instance over an abstract
AF_UNIX socket, which would then allow said process to modify the file
in the store after it has been marked as “valid”.
Vulnerability discovered by puck <https://github.com/puckipedia>.
Nix security advisory:
https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37
Nix fix:
244f3eee0b
* nix/libutil/util.cc (readDirectory): Add variants that take a DIR* and
a file descriptor. Rewrite the ‘Path’ variant accordingly.
(copyFile, copyFileRecursively): New functions.
* nix/libutil/util.hh (copyFileRecursively): New declaration.
* nix/libstore/build.cc (DerivationGoal::buildDone): When ‘fixedOutput’
is true, call ‘copyFileRecursively’ followed by ‘rename’ on each output.
Change-Id: I7952d41093eed26e123e38c14a4c1424be1ce1c4
Reported-by: Picnoir <picnoir@alternativebit.fr>, Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
Change-Id: Idb5f2757f35af86b032a9851cecb19b70227bd88
* nix/libstore/build.cc (SubstitutionGoal::finished): Don’t show what
the empty status looks like.
Change-Id: Ie898432aeb047aff3d59024de6ed6d18f68903c4
Drop the confusing ‘invalid’ jargon and display a hint like we do
for ‘--fallback’.
* nix/libstore/build.cc (DerivationGoal::outputsSubstituted): Rewrite error message.
DerivationGoal::startBuilder() is waiting for an empty line as a check that
the environment set up is fine.
Fixes <https://issues.guix.gnu.org/55324>.
* nix/libstore/build.cc (DerivationGoal::runChild): Remove 'debug'
statement corresponding to bind mounts.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://issues.guix.gnu.org/46212>.
Reported by Christopher Baines <mail@cbaines.net>.
Previously, the nar size returned by 'guix substitute' would be read as
an 'int'; thus, values above 2^31 - 1 would be read and then stored as
negative integers in the database.
Regression introduced in 9dfa20a22a.
* nix/libstore/build.cc (SubstitutionGoal::finished): Use templatized
'string2Int' instead of 'std::atoi' to get an 'unsigned long long',
which is the type of 'hash.second'.
* tests/store.scm ("substitute and large size"): New test.
This avoids the situation where error messages would unintentionally go
to stderr and be wrongfully interpreted as a reply by the daemon.
Fixes <https://bugs.gnu.org/46362>.
This is a followup to ee3226e9d5.
* guix/scripts/substitute.scm (display-narinfo-data): Add 'port'
parameter and honor it.
(process-query): Likewise.
(process-substitution): Likewise.
(%error-to-file-descriptor-4?, with-redirected-error-port): Remove.
(%reply-file-descriptor): New variable.
(guix-substitute): Remove use of 'with-redirected-error-port'. Define
'reply-port' and pass it to 'process-query' and 'process-substitution'.
* nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap
'builderOut' and 'fromAgent'.
* nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter):
Likewise.
* tests/substitute.scm <top level>: Set '%reply-file-descriptor'
rather than '%error-to-file-descriptor-4?'.
Fixes <https://bugs.gnu.org/47229>.
Reported by Nathan Nye of WhiteBeam Security.
* nix/libstore/build.cc (DerivationGoal::startBuilder): When 'useChroot'
is true, add "/top" to 'tmpDir'.
(DerivationGoal::deleteTmpDir): Adjust accordingly. When
'settings.keepFailed' is true, chown in two steps: first the "/top"
sub-directory, and then rename "/top" to its parent.
This removes the main source of latency between subsequent downloads.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Add a
"deduplicate" key to ENV.
(SubstitutionGoal::finished): Remove call to 'optimisePath'.
* guix/scripts/substitute.scm (process-substitution)[destination-in-store?]
[dump-file/deduplicate*]: New variables.
Pass #:dump-file to 'restore-file'.
* guix/scripts/substitute.scm (guix-substitute)[deduplicate?]: New
variable.
Pass #:deduplicate? to 'process-substitution'.
* guix/serialization.scm (dump-file): Export and augment 'dump-file'.
'guix substitute' now takes care of it via 'restore-file'.
* nix/libstore/build.cc (SubstitutionGoal::finished): Remove call to
'canonicalisePathMetaData'.
This way, the hash of the store item can be computed as it is restored,
thereby avoiding an additional file tree traversal ('hashPath' call)
later on in the daemon. Consequently, it should reduce latency between
subsequent substitute downloads.
This is a followup to 5ff521452b.
* guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New
procedure.
(process-substitution): Wrap INPUT into a hash input port, 'hashed', and
read from it. Compare the actual and expected hashes, and print a
"hash-mismatch" status line when they differ. When they match, print
not just "success" but also the nar hash and size.
* nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]:
Remove.
(SubstitutionGoal::finished): Tokenize 'status'. Parse it and handle
"success" and "hash-mismatch" accordingly. Call 'hashPath' only when
the returned hash is not SHA256.
(SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr'
handling.
* tests/substitute.scm ("substitute, invalid hash"): Rename to...
("substitute, invalid narinfo hash"): ... this.
("substitute, invalid hash"): New test.
It was already impossible in practice for 'expectedHashStr' to be empty
if 'status' == "success".
* nix/libstore/build.cc (SubstitutionGoal::finished): Throw 'SubstError'
when 'expectedHashStr' is empty.
That way, when fetching a series of substitutes from the same server(s),
the connection is reused instead of being closed/opened for each
substitutes, which saves on network round trips and TLS handshakes.
* guix/http-client.scm (http-fetch): Add #:keep-alive? and honor it.
* guix/progress.scm (progress-report-port): Add #:close? parameter and
honor it.
* guix/scripts/substitute.scm (at-most): Return the tail as a second
value.
(fetch): Add #:port and #:keep-alive? and honor them.
(%max-cached-connections): New variable.
(open-connection-for-uri/cached, call-with-cached-connection): New
procedures.
(with-cached-connection): New macro.
(process-substitution): Wrap 'fetch' call in 'with-cached-connection'.
Pass #:close? to 'progress-report-port'.
This avoids spawning one substitute process per substitution.
* nix/libstore/build.cc (class Worker)[substituter]: New field.
[outPipe, logPipe, pid]: Remove.
(class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields.
(SubstitutionGoal::timedOut): Adjust to check 'substituter'.
(SubstitutionGoal::tryToRun): Remove references to 'outPipe' and
'logPipe'. Run "guix substitute --substitute" as an 'Agent'. Send the
request with 'writeLine'.
(SubstitutionGoal::finished): Likewise.
(SubstitutionGoal::handleChildOutput): Change to fill in
'expectedHashStr' and 'status'.
(SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally.
(SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'.
* guix/scripts/substitute.scm (process-substitution): Write "success\n"
to stdout upon success.
(%error-to-file-descriptor-4?): New variable.
(guix-substitute): Set 'current-error-port' to file descriptor 4
unless (%error-to-file-descriptor-4?) is false.
Remove "--substitute" arguments. Loop reading line from stdin.
* tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'.
(request-substitution): New procedure.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key")
("substitute, authorized key")
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, two invalid narinfos")
("substitute, narinfo with several URLs"): Adjust to new "guix
substitute --substitute" calling convention.
* nix/libstore/local-store.hh (RunningSubstituter): Remove.
(LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>.
[setSubstituterEnv, didSetSubstituterEnv]: Remove.
[getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove
reference to 'runningSubstituter'.
(LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove.
(LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'.
(LocalStore::querySubstitutablePaths): Spawn substituter agent if
needed. Adjust to 'Agent' interface.
(LocalStore::querySubstitutablePathInfos): Likewise.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to
'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead.
(SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'.
* guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?):
Remove second argument to 'make-parameter'.
(process-query): Call 'warn-about-missing-authentication'
when (%allow-unauthenticated-substitutes?) is #t.
(guix-substitute): Wrap body in 'parameterize'. Set 'guix-warning-port'
too. No longer exit when 'substitute-urls' returns the empty list. No
longer print newline initially.
* tests/substitute.scm (test-quit): Parameterize 'current-error-port' to
account for the port changes in 'guix-substitute'.
