This follows clarification with upstream on the valid name to use:
<https://lists.gnu.org/archive/html/automake/2026-03/msg00005.html>. The
change was made in the recent commit ef4ba3191f.
* build-aux/test-driver.scm (test-runner-gnu): Revert global test result key
to ':global-test-result:'.
Change-Id: If0eca05b8b57f1fd5d848e82bc789c43640d3caf
Test groups were not used in any meaningful way. The group path was not
printed and it was not used in test selection mechanism. I think groups are
useful, and it is nice to be able to, for example, run tests from a single
group.
This commit does two things. First, it changes the test reporting to include
the value returned from test-runner-group-path, so you will know not only the
test name, but the test group(s) as well. And second, it changes the test
selection (and exclusion) process to match against the "full" test name, so
group path + test name.
Hence
(test-begin "failing tests")
(test-equal "this should fail" 1 2)
(test-end)
will, depending on the output location, produce following text.
.trs:
:test-result: FAIL failing tests: this should fail [0.000s]
:test-global-result: FAIL
:recheck: yes
:copy-in-global-log: yes
.log:
test-name: failing tests: this should fail
location: test.scm:140
source:
+ (test-equal "this should fail" 1 2)
expected-value: 1
actual-value: 2
result: FAIL
stdout:
FAIL: test.scm - failing tests: this should fail [0.000s]
* build-aux/test-driver.scm (current-test-full-name): New procedure.
(test-runner-gnu): Use current-test-full-name instead of
test-runner-test-name.
(test-match-name*): Match against current-test-full-name. Use compose.
(test-match-name*/negated): Rewrite in terms of test-match-name*.
Change-Id: I3fb9a2a721165204f020b79e019533f799b790e4
Signed-off-by: Maxim Cournoyer <maxim@guixotic.coop>
Modified-by: Maxim Cournoyer <maxim@guixotic.coop>
The :test-global-result: .trs metadata contained just either FAIL, SKIP or
PASS with a comment that further refinements are required for XPASS. The
description of :test-global-result: is in the manual is as follows:
This is used to declare the "global result" of the script.
Currently, the value of this field is needed only to be reported
(more or less verbatim) in the generated global log file
‘$(TEST_SUITE_LOG)’, so it’s quite free-form. For example, a test
script which runs 10 test cases, 6 of which pass and 4 of which are
skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
while a test script which runs 19 successful tests and one failed
test could have an ‘ALMOST PASSED’ value.
As we can see, the examples as `PASS/SKIP' and `ALMOST PASSED', so there is no
need to stick to strict model. Hence this commit changes the resulting value
to be comma-separated list of PASS, FAIL, XPASS, XFAIL and SKIP. The
respective elements are present only when the count of tests with such a
result is positive.
In practice, that should usually produce lines such as
:test-global-result: PASS,FAIL
or
:test-global-result: PASS
* build-aux/test-driver.scm (test-runner-gnu)[finalize]: Refine the output of
:test-global-result:.
Change-Id: I7178ac9703e1749adf6de2445f7ed0591983cef2
Signed-off-by: Maxim Cournoyer <maxim@guixotic.coop>
Current implementation printed metadata supposed to be present just once per
.trs file on the end of each test group. According to the automake's manual
that is undefined behavior. This commit fixes it by printing that metadata
just once, after all tests did run.
Since there is no built-in hook that could be used for
that (test-runner-on-final runs on *each* outermost test-end), I introduced
new `finalize' procedure that need to be called by the user. Possibly not the
most elegant solution, but since we are the only user, it works fine and
produces actually valid .trs file.
That also means there is no longer any use for test-runner-on-test-end!.
* build-aux/test-driver.scm (test-runner-gnu): Define new procedure `finalize'
and return it together with the runner. Do not call
test-runner-on-group-end!.
(main): Call the `finalize' after all tests are done.
Signed-off-by: Maxim Cournoyer <maxim@guixotic.coop>
Guile 3.0.11 ships with a new SRFI 64.
* build-aux/test-driver.scm (%test-match-all): Delete procedure.
(main): Replace %test-match-all with test-match-all.
Change-Id: Ibcd7d98194694c1de679491be4f5a603a79f6e29
With srfi-64 from guile 3.0.11 the test name is no longer in the
test-result-alist.
* build-aux/test-driver.scm (test-runner-gnu): Use test-runner-test-name to
access test-name.
Change-Id: I53d0fdd8db0d1af5e636e3f2a68280cd2bddfe4c
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Merges: #5836
Signed-off-by: Rutherther <rutherther@ditigal.xyz>
This reverts commit a7db92d9b3, this time with
the more careful command, to avoid a world rebuild:
grep -rl --exclude-dir=build --exclude-dir=patches --exclude=ld-wrapper.in \
--exclude=sitecustomize.py --exclude=guix-emacs.el maxim.cournoyer@gmail.com |
xargs sed -i 's/maxim.cournoyer@gmail.com/maxim@guixotic.coop/g'
Change-Id: I0871f5cf9ace79e34e78e24154d0da6ea95a3a91
The change was automated via:
git grep -l maxim.cournoyer@gmail.com |
xargs sed -i 's/maxim.cournoyer@gmail.com/maxim@guixotic.coop/g'
* .mailmap: New entry.
Change-Id: Iaa88b1e51c94159c49138fb43910badb990eb169
Emacs auto-detects the build-aux/test-driver.scm as sh-mode, due to the
shebang. So override the auto-detection using a local variable forcing
scheme-mode.
* build-aux/test-driver.scm: Set mode to scheme.
Change-Id: I07f12656dba5fb7b52447b94569307a2a605ffc1
Signed-off-by: Maxim Cournoyer <maxim.cournoyer@gmail.com>
* build-aux/test-driver.scm (script-version): Update.
(show-help): Document it.
(%options): Add the 'show-duration' option.
(test-runner-gnu): Pass as a new argument.
[test-cases-start-time]: New inner variable.
[test-on-test-begin-gnu]: New hook, used to record the start time.
[test-on-test-end-gnu]: Conditionally print elapsed time. Record it as the
optional metadata in the test result file (.trs).
* doc/guix.texi (Running the Test Suite): Document it.
* build-aux/test-driver.scm: Add an exec-based shebang and set the script
executable bit.
(main): Insert a newline after the version string is printed with --version.
* build-aux/test-driver.scm (show-help): Add the help text for the
new '--errors-only' option.
(%options): Add the errors-only option.
(test-runner-gnu): Add the errors-only? parameter and update doc. Move the
logging of the test data after the test has completed, so a choice can be made
whether to keep it or discard it based on the value of the test result.
(main): Pass the errors-only? option to the driver.
* doc/guix.texi (Running the Test Suite): Document the new option.
* build-aux/test-driver.scm (show-help): Add help text for the new --select
and --exclude options.
(%options): Add the new select and exclude options.
(test-runner-gnu): Pass them to the test runner. Update doc.
(test-match-name*, test-match-name*/negated, %test-match-all): New variables.
(main): Compute the test specifier based on the values of the new options and
apply it to the current test runner when running the test file.
* doc/guix.texi (Running the Test Suite): Document the new options.
The Automake parallel test harness does its own smart detection of the
terminal color capability and always provides the --color-tests argument to
the driver. This change defaults the --color-tests argument to true when the
test driver is run on its own (not via Automake).
* build-aux/test-driver.scm (main): Set the default value of the --color-tests
argument to true when it's not explicitly provided.
This makes it easier (and less surprising) for users to experiment with the
custom Scheme test driver directly. The behavior is unchanged from Automake's
point of view.
* build-aux/test-driver.scm (main): Make the --log-file and --trs-file
arguments optional and update doc. Only open, redirect and close a port to a log file when
the --log-file option is provided. Only open and close a port to a trs file
when the --trs-file option is provided.
(test-runner-gnu): Set OUT-PORT parameter default value to the current output
port. Set the TRS-PORT parameter default value to a void port. Update doc.
Previously it would print them with 'display', which was inconvenient.
* build-aux/test-driver.scm (test-display): Use ~S for VALUE.
(test-runner-gnu): Use 'format' instead of 'test-display' for the test
name, location, and result kind.
Before that '.log' files for scheme tests were fragmented and not
included in test-suite.log. This unifies the semantics of SRFI-64 API
with Automake test suite.
* build-aux/test-driver.scm: New file.
* Makefile.am (SCM_LOG_DRIVER, AM_SCM_LOG_DRIVER_FLAGS): New variables.
(SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables.
(AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0.
* test-env.in: Silence guix-daemon.
* doc/guix.texi (Running the Test Suite): Describe how to display the
detailed results. Bug reports require only 'test-suite.log' file.
* tests/base32.scm, tests/build-utils.scm, tests/builders.scm,
tests/challenge.scm, tests/cpan.scm, tests/cpio.scm, tests/cran.scm,
tests/cve.scm, tests/derivations.scm, tests/elpa.scm,
tests/file-systems.scm, tests/gem.scm, tests/gexp.scm,
tests/gnu-maintenance.scm, tests/grafts.scm, tests/graph.scm,
tests/gremlin.scm, tests/hackage.scm, tests/hash.scm,
tests/import-utils.scm, tests/lint.scm, tests/monads.scm, tests/nar.scm,
tests/packages.scm, tests/pk-crypto.scm, tests/pki.scm,
tests/profiles.scm, tests/publish.scm, tests/pypi.scm,
tests/records.scm, tests/scripts-build.scm, tests/scripts.scm,
tests/services.scm, tests/sets.scm, tests/size.scm, tests/snix.scm,
tests/store.scm, tests/substitute.scm, tests/syscalls.scm,
tests/system.scm, tests/ui.scm, tests/union.scm, tests/upstream.scm,
tests/utils.scm: Don't exit at the end of test groups.
* tests/containers.scm: Likewise. Use 'test-skip' instead of exiting
with error code 77.
