Skip to content

@setup scenarios run even when Feature is skipped #2169

@edwardsph

Description

@edwardsph

I discovered this in 1.3.0RC2 minutes after 1.3.0 was released and have upgraded to confirm it still an issue.

I created a feature:

@test
Feature: sample karate test script
  Background:
    * url 'https://jsonplaceholder.typicode.com'

  @setup
  Scenario: setup
    * table tests
      | test    |
      | 'users' |

  Scenario Outline: get all <test>
    Given path '<test>'
    When method get
    Then status 200

    Examples:
      | karate.setup().tests |

If I run this without skipping any tags it reports:

features:     1 | skipped:    0 | efficiency: 0.05
scenarios:    2 | passed:     2 | failed: 0

image

Problem 1: Ideally a @setup scenario would not be included in the counts as it is misleading. It is only run for the benefit of the other scenario.

If I then run with .tags("~@test") it reports:

features:     1 | skipped:    0 | efficiency: 0.00
scenarios:    1 | passed:     1 | failed: 0

image

Problem 2: This is even more misleading since I wouldn't want any scenarios to count, nor should the feature be counted

If I convert it to a standard scenario with no @setup and run it with the tag skipped the results are:

features:     0 | skipped:    1 | efficiency: 0.00
scenarios:    0 | passed:     0 | failed: 0

image

This is what I would expect in the second case.

I think it makes sense to show the @setup scenario results in the report but it makes less sense to have it included in test results counts and to impact feature counts in the way it does.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions