Print the bash command that caused the test to fail #15

ryansouza · 2013-07-03T03:12:33Z

Includes the $BASH_COMMAND and exit status that caused the test failure in the pretty failure output

$ cat example.bats 
@test "a passing test" {
  true
}

@test "a failing test" {
  false
}

@test "another failing test" {
  [ true == false ]
}

@test "a failing test with variables" {
  X=1
  [ $X -eq 2 ]
}

$ bin/bats example.bats 
 ✓ a passing test
 ✗ a failing test
   (in test file /Users/ryansouza/p/bats/example.bats, line 6)
     'false' exited with status 1
 ✗ another failing test
   (in test file /Users/ryansouza/p/bats/example.bats, line 10)
     '[ true == false ]' exited with status 1
 ✗ a failing test with variables
   (in test file /Users/ryansouza/p/bats/example.bats, line 15)
     '[ $X -eq 2 ]' exited with status 1

4 tests, 3 failures

$ bin/bats -t example.bats 
1..4
ok 1 a passing test
not ok 2 a failing test
# (in test file /Users/ryansouza/p/bats/example.bats, line 6)
#   'false' exited with status 1
not ok 3 another failing test
# (in test file /Users/ryansouza/p/bats/example.bats, line 10)
#   '[ true == false ]' exited with status 1
not ok 4 a failing test with variables
# (in test file /Users/ryansouza/p/bats/example.bats, line 15)
#   '[ $X -eq 2 ]' exited with status 1

edit: rebased & updated (11/10)

ryansouza · 2013-07-03T03:15:47Z

Tests pass when run in-repo with bin/bats, but fail with a bats installed with homebrew. Not sure whats up with that

~/p/bats(failure-context)$ bats test
1..21
ok 1 no arguments prints usage instructions
ok 2 -v and --version print version number
ok 3 invalid filename prints an error
ok 4 empty test file runs zero tests
ok 5 one passing test
not ok 6 one failing test
    /var/folders/5m/1d2bfcd114j3dmxf9zb7q0v80000gn/T/bats.88641.src: line 42: [: =: unary operator expected
not ok 7 one failing and one passing test
ok 8 test environments are isolated
ok 9 setup is run once before each test
ok 10 teardown is run once after each test, even if it fails
ok 11 load sources scripts relative to the current test file
ok 12 load aborts if the specified script does not exist
not ok 13 output is discarded for passing tests and printed for failing tests
ok 14 -c prints the number of tests
not ok 15 dash-e is not mangled on beginning of line
not ok 16 dos line endings are stripped before testing
ok 17 running a suite with no test files
ok 18 running a suite with one test file
ok 19 counting tests in a suite
ok 20 aggregated output of multiple tests in a suite
ok 21 a failing test in a suite results in an error exit code

~/p/bats(failure-context)$ bin/bats test
1..21
ok 1 no arguments prints usage instructions
ok 2 -v and --version print version number
ok 3 invalid filename prints an error
ok 4 empty test file runs zero tests
ok 5 one passing test
ok 6 one failing test
ok 7 one failing and one passing test
ok 8 test environments are isolated
ok 9 setup is run once before each test
ok 10 teardown is run once after each test, even if it fails
ok 11 load sources scripts relative to the current test file
ok 12 load aborts if the specified script does not exist
ok 13 output is discarded for passing tests and printed for failing tests
ok 14 -c prints the number of tests
ok 15 dash-e is not mangled on beginning of line
ok 16 dos line endings are stripped before testing
ok 17 running a suite with no test files
ok 18 running a suite with one test file
ok 19 counting tests in a suite
ok 20 aggregated output of multiple tests in a suite
ok 21 a failing test in a suite results in an error exit code

ryansouza · 2013-07-03T18:34:16Z

Would be cool if there was a way to see the command with variables expanded, not sure if thats possible though. Right now it looks like

not ok 5 bats error trap shows expanded vars?       
# /opt/busser/suites/bats/verify_app_is_running.bats:1       
#   [ 1 -eq $X ] exited with status 1

AnneTheAgile · 2013-10-06T18:42:20Z

This looks helpful! I hope the pull is considered.

lzap · 2013-10-30T19:28:08Z

Maybe make this optional via new option?

sstephenson · 2013-10-30T20:48:11Z

@ryansouza I like this patch. Would you be interested in rebasing it against the changes in master?

AnneTheAgile · 2013-10-31T01:18:24Z

In case @lzap 's idea of an option is desired, it seems we're a bit limited on the best letter to use, since Bats only parses the first letter, ie;
https://github.com/sstephenson/bats/blob/bfa4ebcd0f5b75addedac3361328f73416d1c274/libexec/bats

The letters c, h, p, t, v are used. Thus it would be ambiguous to use v=verbosity (used for version), c for command (used for count), t for trace (used for tap).

Perhaps letter 'd' could be used, show 'debug' information?

Refs,
[]Nose has both -V for version and -v for verbosity. https://nose.readthedocs.org/en/latest/usage.html#extended-usage
[]Many options here; http://ftp.gnu.org/pub/old-gnu/Manuals/ld-2.9.1/html_node/ld_3.html

mislav · 2013-10-31T07:33:15Z

I would love this feature. It would replace my helper that I use for test suites:

assert() {
  if ! "$@"; then
    flunk "failed: $@"
  fi
}

(flunk is also a custom helper method)

lzap · 2013-10-31T10:24:17Z

Speaking about options and letters, I started different shell project some time ago and quickly ran out of letters. So I had to refactor it a bit to use external GNU getopt which works with both short and long options for checking options and pure Bash for parsing them. Just for some inspiration (not saying this is good example of how to write shell scripts :-)

https://github.com/lzap/snap-guest/blob/master/snap-guest

sstephenson · 2013-11-10T22:37:44Z

Let's keep it simple. We don't need to hide this behind a command-line option.

ryansouza · 2013-11-10T22:41:13Z

I'll give the rebase a try when I get home

ryansouza · 2013-11-11T00:40:39Z

Rebased, updated the pull description with a better example & pretty output

sstephenson · 2013-11-11T16:16:41Z

This is looking good. It doesn't properly display failing commands in teardown functions, though. Here's a minimal test case:

teardown() {
  false
}

@test "truth" {
  true
}

 ✗ truth
   (from function `teardown' in test file /Users/sam/Desktop/test.bats, line 2)
     '' exited with status

ryansouza · 2013-11-11T20:29:25Z

This is proving to be a tough bug to squash, definitely flexing my bash debug skills.

Appears to be calling teardown in an if statement is swallowing the err trap.

Easy way out: just not display that line if theres no BATS_FAILED_COMMAND. I'll keep poking at it

b-long · 2014-05-12T13:43:04Z

Any update on this? Seems like a very useful feature to add to bats.

sstephenson · 2014-05-12T14:36:45Z

I haven't had a chance to work on this, but if we can resolve the issue I described above, I'll be happy to merge it.

b-long · 2014-05-12T15:08:54Z

Fair enough, thanks @sstephenson . I don't have time today, but if I do soon, I'll try and update the teardown function.

ryansouza · 2014-05-12T18:28:08Z

Yeah I spent some time trying to get it to work with no luck

Didn't realize its been 6 months though, oof :\

sstephenson · 2014-06-02T01:11:20Z

Closing in favor of #57

Print the bash command that caused the test to fail

57d0d76

This was referenced May 21, 2014

Run with stdout and stderr separated, load by absolute path, print failed assertion on error. #53

Closed

Print failed statement in backtrace #57

Merged

sstephenson closed this Jun 2, 2014

Print the bash command that caused the test to fail #15

Print the bash command that caused the test to fail #15

Uh oh!

Conversation

ryansouza commented Jul 3, 2013

Uh oh!

ryansouza commented Jul 3, 2013

Uh oh!

ryansouza commented Jul 3, 2013

Uh oh!

AnneTheAgile commented Oct 6, 2013

Uh oh!

lzap commented Oct 30, 2013

Uh oh!

sstephenson commented Oct 30, 2013

Uh oh!

AnneTheAgile commented Oct 31, 2013

Uh oh!

mislav commented Oct 31, 2013

Uh oh!

lzap commented Oct 31, 2013

Uh oh!

sstephenson commented Nov 10, 2013

Uh oh!

ryansouza commented Nov 10, 2013

Uh oh!

ryansouza commented Nov 11, 2013

Uh oh!

sstephenson commented Nov 11, 2013

Uh oh!

ryansouza commented Nov 11, 2013

Uh oh!

b-long commented May 12, 2014

Uh oh!

sstephenson commented May 12, 2014

Uh oh!

b-long commented May 12, 2014

Uh oh!

ryansouza commented May 12, 2014

Uh oh!

sstephenson commented Jun 2, 2014

Uh oh!

Uh oh!