-
-
Notifications
You must be signed in to change notification settings - Fork 2k
Description
I discovered this in 1.3.0RC2 minutes after 1.3.0 was released and have upgraded to confirm it still an issue.
I created a feature:
@test
Feature: sample karate test script
Background:
* url 'https://jsonplaceholder.typicode.com'
@setup
Scenario: setup
* table tests
| test |
| 'users' |
Scenario Outline: get all <test>
Given path '<test>'
When method get
Then status 200
Examples:
| karate.setup().tests |
If I run this without skipping any tags it reports:
features: 1 | skipped: 0 | efficiency: 0.05
scenarios: 2 | passed: 2 | failed: 0
Problem 1: Ideally a @setup
scenario would not be included in the counts as it is misleading. It is only run for the benefit of the other scenario.
If I then run with .tags("~@test")
it reports:
features: 1 | skipped: 0 | efficiency: 0.00
scenarios: 1 | passed: 1 | failed: 0
Problem 2: This is even more misleading since I wouldn't want any scenarios to count, nor should the feature be counted
If I convert it to a standard scenario with no @setup
and run it with the tag skipped the results are:
features: 0 | skipped: 1 | efficiency: 0.00
scenarios: 0 | passed: 0 | failed: 0
This is what I would expect in the second case.
I think it makes sense to show the @setup
scenario results in the report but it makes less sense to have it included in test results counts and to impact feature counts in the way it does.