When reusing a HTTP connection to fetch multiple nars, and the remote server
signals that the connection should be closed.
* guix/substitutes.scm (download-nar): Close connections to substitute servers
when a Connection: close header is specified in the response.
Change-Id: Id3746c848a2157419060d5d968f724bc82a28e49
To be consistent with other procedures that make network requests.
* guix/substitutes.scm (download-nar): Add #:keep-alive? option.
* guix/scripts/substitute.scm (process-substitution/fallback)
(process-substitution): Call download-nar with #:keep-alive? #t.
Change-Id: I83b27d0c3a0916d058fbbbeb7aa77dbb8a742768
From the substitutes script. This makes it possible to use download-nar in
the the Guile guix-daemon.
* guix/scripts/substitute.scm (%fetch-timeout): Move down to where it's now
used.
(%random-state, with-timeout, catch-system-error, http-response-error?,
download-nar): Move to…
* guix/substitutes.scm: …here.
Change-Id: I8c09bf4b33cb5c6d042057d4d9adeb36c24c11dc
Fixes <https://issues.guix.gnu.org/77606>.
Calling ‘fdatasync’ for each and every narinfo file created by ‘guix
substitute’ proved to be too expensive on spinning HDDs and/or under
load (from 0.1s to 1.3s for the ‘fdatasync’ call alone).
* guix/git-authenticate.scm (cache-authenticated-commit): Pass #:sync? #f.
* guix/http-client.scm (http-fetch/cached): Likewise.
* guix/scripts/discover.scm (write-publish-file): Likewise.
* guix/scripts/style.scm (format-whole-file): Likewise.
* guix/substitutes.scm (cache-narinfo!): Likewise.
Reported-by: Christopher Baines <mail@cbaines.net>
Change-Id: I82297eae737bc5aae8a3f7604119e9f3d4b625bf
Many thanks to Reepca Russelstein for their review and guidance on these
changes.
* nix/libstore/build.cc (guestUID, guestGID): New variables.
(DerivationGoal)[readiness]: New field.
(initializeUserNamespace): New function.
(DerivationGoal::runChild): When ‘readiness.readSide’ is positive, read
from it.
(DerivationGoal::startBuilder): Call ‘chown’
only when ‘buildUser.enabled()’ is true. Pass CLONE_NEWUSER to ‘clone’
when ‘buildUser.enabled()’ is false or not running as root. Retry
‘clone’ without CLONE_NEWUSER upon EPERM.
(DerivationGoal::registerOutputs): Make ‘actualPath’ writable before
‘rename’.
(DerivationGoal::deleteTmpDir): Catch ‘SysError’ around ‘_chown’ call.
* nix/libstore/local-store.cc (LocalStore::createUser): Do nothing if
‘dirs’ already exists. Warn instead of failing when failing to chown
‘dir’.
* guix/substitutes.scm (%narinfo-cache-directory): Check for
‘_NIX_OPTIONS’ rather than getuid() == 0 to determine the cache
location.
* doc/guix.texi (Build Environment Setup): Reorganize a bit. Add
section headings “Daemon Running as Root” and “The Isolated Build
Environment”. Add “Daemon Running Without Privileges” subsection.
Remove paragraph about ‘--disable-chroot’.
(Invoking guix-daemon): Warn against ‘--disable-chroot’ and explain why.
* tests/derivations.scm ("builder is outside the store"): New test.
Reviewed-by: Reepca Russelstein <reepca@russelstein.xyz>
Fixes <https://issues.guix.gnu.org/75679>.
Fixes this infamous error:
substitute: looking for substitutes on 'https://bordeaux.guix.gnu.org'... 0.0%guix substitute: error: TLS error in procedure 'write_to_session_record_port': Error in the push function.
guix system: error: `/gnu/store/…-guix-1.4.0-32.5f6fdad/bin/guix substitute' died unexpectedly
* guix/substitutes.scm (call-with-connection-error-handling): Report
‘error/push-error’ and ‘error/pull-error’ as mere warnings.
Reported-by: Timo Wilken <git@twilken.net>
Change-Id: Ibc507cb88b164cbb91dff95aa18d9974c573f7c9
Previous values were overly conservative and often an annoyance.
* guix/substitutes.scm (%narinfo-negative-ttl): Reduce to 2mn.
(%narinfo-transient-error-ttl): Reduce to 1mn.
Change-Id: I8cff927a15d8203fb370369a56f024b8a14f3cc3
This was obtained by setting up this environment:
guix shell -D guix --with-input=guile@3.0.9=guile-next \
--with-commit=guile-next=e2ed33ef0445c867fe56c247054aa67e834861f2
-- make -j5
then adding 'unused-module' to (@@ (guix build compiler) %warnings),
building, and checking all the "unused module" warnings and removing
those that were definitely unused.
Fixes <https://issues.guix.gnu.org/51441>.
Reported by Z572 <873216071@qq.com> and Peng Mei Yu <pengmeiyu@riseup.net>.
Previously, passing '--substitute-urls=https://mirror.sjtu.edu.cn/guix',
without a trailing slash, would fail due to incorrectly constructed
narinfo URLs. Users would have to explicitly add a trailing slash.
* guix/substitutes.scm (narinfo-request): Ensure BASE has a trailing
slash.
Use relative URIs and "resolve-uri-reference" (which implements the algorithm
specified in RFC 3986 section 5.2.2) for building the URL, instead of just
appending strings. This avoids issued if the cache-url ends with a slash.
* guix/substitutes.scm (narinfo-request): Use resolve-uri-reference for
constructing the url.
As an additional value, in addition to the narinfos. This value is useful in
the weather script for reporting how many requests to the substitute server
were made.
* guix/substitutes.scm (lookup-narinfos): Additionally return the number of
requests made.
Rather than always outputting to (current-error-port) in
lookup-narinfos (which is called from within lookup-narinfos/diverse), take a
procedure which should return a progress reporter, and defer any output to
that.
As this is now general purpose code, make the default behaviour to output
nothing. Maintain the current behaviour of the substitute script by moving the
progress reporter implementation there, and passing it in when calling
lookup-narinfos/diverse.
These changes should be generally useful, but I'm particularly looking at
getting guix weather to do progress reporting differently, with this new
flexibility.
* guix/substitutes.scm (fetch-narinfos): Take a procedure to make a
progress-reporter, and use that rather than the hardcoded behaviour.
(lookup-narinfos): Add #:make-progress-reporter keyword argument, and pass
this through to fetch-narinfos.
(lookup-narinfos/diverse): Add a #:make-progress-reporter keyword argument,
and pass this through to lookup-narinfos.
* guix/scripts/substitute.scm (process-query): Pass a progress-reporter to
lookup-narinfos/diverse.
This means there's a module for working with substitutes, rather than all the
code sitting in the script. The need for this can be seen with the weather and
challenge scripts, that now don't have to use code from the substitute script,
but can instead use the substitute module.
The separation here between the actual functionality of the substitute script
and the underlying functionality used both there and elsewhere should make
maintenance easier moving forward.
This commit just moves code, none of the code should have been changed
significantly.
* guix/scripts/substitute.scm (%narinfo-cache-directory, %narinfo-ttl,
%narinfo-negative-ttl, %narinfo-transient-error-ttl, %unreachable-hosts): Move
variables to guix/substitutes.scm.
(narinfo-cache-file, cached-narinfo, cache-narinfo!, narinfo-request,
read-to-eof, call-with-connection-error-handling, fetch-narinfos,
lookup-narinfos, lookup-narinfos/diverse): Move procedures to
guix/substitutes.scm.
* guix/substitutes.scm: New file.
* Makefile.am: Add it.
* guix/narinfo.scm: Remove redundant module.
* guix/scripts/challenge.scm: Change (guix scripts substitute) to (guix
substitutes).
* guix/scripts/weather.scm: Change (guix scripts substitute) to (guix
substitutes).
Christopher Baines