Skip to content

Run's =<exit-status> syntax is still documented #578

@jpluscplusm

Description

@jpluscplusm

Describe the bug

The run =<expected-exit-status> convenience syntax (documented here) is broken, and does not work.

I believe anyone relying on the currently-documented syntax to assert a run command's exit status has been inadvertently testing bats' breakage, not their underlying command's success or failure.

To Reproduce

Running bats with the attached test file below should produce 4 test failures, according to the docs. Instead, it produces 2 passes and 2 failures. The correctly-failing-failures demonstrate that the alternative run -<expected-exit-status> syntax does work as expected.

$ bats foo.bats 
foo.bats
 ✓ This test should fail, but will succeed, #1
   ## Command that 'run' ran was: '=0 false'
 ✓ This test should fail, but will succeed, #2
   ## Command that 'run' ran was: '=1 true'
 ✗ This test should fail, and will fail, #1
   (in test file foo.bats, line 25)
     `run -0 false' failed, expected exit code 0, got 1
 ✗ This test should fail, and will fail, #2
   (in test file foo.bats, line 29)
     `run -1 true' failed, expected exit code 1, got 0

4 tests, 2 failures

Notice that the reporting line that shows BATS_RUN_COMMAND for each of the incorrectly-passing tests demonstrates that the =<exit-status> is injected as a prefix to the actual command the user wanted to test.

And because (to a reasonable approximation) ~100.00% of machines lack a command named =0/=1/etc, the fact that run (by design) masks the exit code of the erroneously-missing =0/=1 command, anyone using this syntax to test either success or failure will not have been testing -- or even executing -- the command they thought they were testing, since #479 was merged in October 2021.

#!/usr/bin/env bats

# Testing with this file should result in 4x failing tests.
# Instead, it will result in 2x failing and 2x passing tests.

report() {
  echo "# ## Command that 'run' ran was: '$BATS_RUN_COMMAND'" >&3
}

# 2x incorrectly passing tests

@test "This test should fail, but will succeed, #1" {
  run =0 false
  report
}
@test "This test should fail, but will succeed, #2" {
  run =1 true
  report
}

# 2x correctly failing tests, exactly the same as the previous 2 tests, but
# with "=<N>" changed to "-<N>"

@test "This test should fail, and will fail, #1" {
  run -0 false
  report
}
@test "This test should fail, and will fail, #2" {
  run -1 true
  report
}

Expected behavior

I expected 4 test failures.

Environment (please complete the following information):

  • Bats Version 1.6.0 @ 5c7a96727535a91fcef50b22783f0a54e10821eb
  • OS:
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
  • Bash version: GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

Additional context

I have opened PR #579 to change the "Writing Tests" documentation to stop pointing users to the old syntax, and to use the current syntax.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions