-
Notifications
You must be signed in to change notification settings - Fork 442
Description
Describe the bug
The run =<expected-exit-status>
convenience syntax (documented here) is broken, and does not work.
I believe anyone relying on the currently-documented syntax to assert a run
command's exit status has been inadvertently testing bats' breakage, not their underlying command's success or failure.
To Reproduce
Running bats
with the attached test file below should produce 4 test failures, according to the docs. Instead, it produces 2 passes and 2 failures. The correctly-failing-failures demonstrate that the alternative run -<expected-exit-status>
syntax does work as expected.
$ bats foo.bats
foo.bats
✓ This test should fail, but will succeed, #1
## Command that 'run' ran was: '=0 false'
✓ This test should fail, but will succeed, #2
## Command that 'run' ran was: '=1 true'
✗ This test should fail, and will fail, #1
(in test file foo.bats, line 25)
`run -0 false' failed, expected exit code 0, got 1
✗ This test should fail, and will fail, #2
(in test file foo.bats, line 29)
`run -1 true' failed, expected exit code 1, got 0
4 tests, 2 failures
Notice that the reporting line that shows BATS_RUN_COMMAND for each of the incorrectly-passing tests demonstrates that the =<exit-status>
is injected as a prefix to the actual command the user wanted to test.
And because (to a reasonable approximation) ~100.00% of machines lack a command named =0
/=1
/etc, the fact that run
(by design) masks the exit code of the erroneously-missing =0
/=1
command, anyone using this syntax to test either success or failure will not have been testing -- or even executing -- the command they thought they were testing, since #479 was merged in October 2021.
#!/usr/bin/env bats
# Testing with this file should result in 4x failing tests.
# Instead, it will result in 2x failing and 2x passing tests.
report() {
echo "# ## Command that 'run' ran was: '$BATS_RUN_COMMAND'" >&3
}
# 2x incorrectly passing tests
@test "This test should fail, but will succeed, #1" {
run =0 false
report
}
@test "This test should fail, but will succeed, #2" {
run =1 true
report
}
# 2x correctly failing tests, exactly the same as the previous 2 tests, but
# with "=<N>" changed to "-<N>"
@test "This test should fail, and will fail, #1" {
run -0 false
report
}
@test "This test should fail, and will fail, #2" {
run -1 true
report
}
Expected behavior
I expected 4 test failures.
Environment (please complete the following information):
- Bats Version 1.6.0 @
5c7a96727535a91fcef50b22783f0a54e10821eb
- OS:
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
- Bash version:
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
Additional context
I have opened PR #579 to change the "Writing Tests" documentation to stop pointing users to the old syntax, and to use the current syntax.