Skip to content

Trainer doesn't recognise failed parametrised tests #29174

New issue
New issue
Closed
Closed
Trainer doesn't recognise failed parametrised tests#29174
@evgeny-sureev

Description

@evgeny-sureev
evgeny-sureev
opened on Dec 17, 2024

New Issue Checklist

Issue Description

If using new Swift Testing framework parametrised tests, trainer parses failures as successes.

Example test file:

import Testing

struct ParametrizedFailsTests {
    @Test
    func testSampleFail() {
        let number = 5
        #expect(number < 3, "Number \(number) is too big")
    }
    
    @Test(arguments: [1,2,3,4,5])
    func testParametrizedFails(_ number: Int) {
        #expect(number < 3, "Number \(number) is too big")
    }
}
Command executed

bundle exec fastlane test

Complete output when running fastlane, including the stack trace and command used
[✔] 🚀
[16:22:08]: ------------------------------
[16:22:08]: --- Step: default_platform ---
[16:22:08]: ------------------------------
[16:22:08]: Driving the lane 'ios test' 🚀
[16:22:08]: -----------------------
[16:22:08]: --- Step: run_tests ---
[16:22:08]: -----------------------
[16:22:08]: Resolving Swift Package Manager dependencies...
[16:22:08]: $ xcodebuild -resolvePackageDependencies -scheme ParametrizedFails -project ./ParametrizedFails.xcodeproj
[16:22:10]: ▸ Command line invocation:
[16:22:10]: ▸     /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies -scheme ParametrizedFails -project ./ParametrizedFails.xcodeproj
[16:22:10]: ▸ User defaults from command line:
[16:22:10]: ▸     IDEPackageSupportUseBuiltinSCM = YES
[16:22:10]: ▸ 2024-12-17 16:22:10.178 xcodebuild[37534:51715732]  DVTDeviceOperation: Encountered a build number "" that is incompatible with DVTBuildVersion.
[16:22:10]: ▸ 2024-12-17 16:22:10.179 xcodebuild[37534:51715670] [MT] DVTDeviceOperation: Encountered a build number "" that is incompatible with DVTBuildVersion.
[16:22:10]: ▸ resolved source packages:
[16:22:10]: $ xcodebuild -showBuildSettings -scheme ParametrizedFails -project ./ParametrizedFails.xcodeproj 2>&1

+--------------------------------------------------------------------------------------------------------------------------------------------+

|                                                          Summary for scan 2.226.0                                                          |

+------------------------------------------------+-------------------------------------------------------------------------------------------+

| scheme                                         | ParametrizedFails                                                                         |

| device                                         | iPhone 16 Pro                                                                             |

| fail_build                                     | false                                                                                     |

| project                                        | ./ParametrizedFails.xcodeproj                                                             |

| skip_detect_devices                            | false                                                                                     |

| ensure_devices_found                           | false                                                                                     |

| force_quit_simulator                           | false                                                                                     |

| reset_simulator                                | false                                                                                     |

| disable_slide_to_type                          | true                                                                                      |

| reinstall_app                                  | false                                                                                     |

| clean                                          | false                                                                                     |

| open_report                                    | false                                                                                     |

| output_directory                               | ./fastlane/test_output                                                                    |

| output_types                                   | html,junit                                                                                |

| buildlog_path                                  | ~/Library/Logs/scan                                                                       |

| include_simulator_logs                         | false                                                                                     |

| xcodebuild_formatter                           | xcpretty                                                                                  |

| output_remove_retry_attempts                   | false                                                                                     |

| derived_data_path                              | /Users/evgeny/Library/Developer/Xcode/DerivedData/ParametrizedFails-gumghbybnfcsafffdduy  |

|                                                | zfglmpwv                                                                                  |

| should_zip_build_products                      | false                                                                                     |

| output_xctestrun                               | false                                                                                     |

| result_bundle                                  | false                                                                                     |

| use_clang_report_name                          | false                                                                                     |

| disable_concurrent_testing                     | false                                                                                     |

| skip_build                                     | false                                                                                     |

| slack_use_webhook_configured_username_and_icon | false                                                                                     |

| slack_username                                 | fastlane                                                                                  |

| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png                                       |

| skip_slack                                     | false                                                                                     |

| slack_only_on_failure                          | false                                                                                     |

| run_rosetta_simulator                          | false                                                                                     |

| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                                                         |

| skip_package_dependencies_resolution           | false                                                                                     |

| disable_package_automatic_updates              | false                                                                                     |

| use_system_scm                                 | false                                                                                     |

| number_of_retries                              | 0                                                                                         |

| xcode_path                                     | /Applications/Xcode.app                                                                   |

+------------------------------------------------+-------------------------------------------------------------------------------------------+


[16:22:12]: Disabling 'Slide to Type' iPhone 16 Pro

[16:22:12]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/evgeny/Library/Developer/CoreSimulator/Devices/9F40E622-7C9C-4210-81B4-C59E0F02B7D7/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1

[16:22:12]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme ParametrizedFails -project ./ParametrizedFails.xcodeproj -derivedDataPath /Users/evgeny/Library/Developer/Xcode/DerivedData/ParametrizedFails-gumghbybnfcsafffdduyzfglmpwv -destination 'platform=iOS Simulator,id=9F40E622-7C9C-4210-81B4-C59E0F02B7D7' build test | tee '/Users/evgeny/Library/Logs/scan/ParametrizedFails-ParametrizedFails.log' | xcpretty  --report html --output '/Users/evgeny/Documents/test/1/ParametrizedFails/fastlane/test_output/report.html' --report junit --output '/Users/evgeny/Documents/test/1/ParametrizedFails/fastlane/test_output/report.junit' --report junit --output '/var/folders/jg/84njmnrx1136dws_knb1xcnc0000gn/T/junit_report20241217-37523-66w3xq'

[16:22:12]: ▸ Loading...

[16:22:12]: ▸ 2024-12-17 16:22:12.887 xcodebuild[37567:51715969]  DVTDeviceOperation: Encountered a build number "" that is incompatible with DVTBuildVersion.

[16:22:12]: ▸ 2024-12-17 16:22:12.890 xcodebuild[37567:51715957] [MT] DVTDeviceOperation: Encountered a build number "" that is incompatible with DVTBuildVersion.

[16:22:12]: ▸ --- xcodebuild: WARNING: Using the first of multiple matching destinations:

[16:22:12]: ▸ { platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

[16:22:12]: ▸ { platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

[16:22:15]: ▸ Processing empty-ParametrizedFails.plist

[16:22:15]: ▸ Linking ParametrizedFails.debug.dylib

[16:22:15]: ▸ Linking ParametrizedFails

[16:22:15]: ▸ Build Succeeded

[16:22:15]: ▸ --- xcodebuild: WARNING: Using the first of multiple matching destinations:

[16:22:15]: ▸ { platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

[16:22:15]: ▸ { platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

[16:22:16]: ▸ Processing empty-ParametrizedFails.plist

[16:22:16]: ▸ Linking ParametrizedFails.debug.dylib

[16:22:16]: ▸ Linking ParametrizedFails

[16:22:16]: ▸ Processing empty-ParametrizedFailsTests.plist

[16:22:16]: ▸ Linking ParametrizedFailsTests

[16:22:16]: Running Tests: ▸ Touching ParametrizedFailsTests.xctest (in target 'ParametrizedFailsTests' from project 'ParametrizedFails')

[16:22:55]: ▸ 2024-12-17 16:22:55.779 xcodebuild[37567:51715957] [MT] IDETestOperationsObserverDebug: 39.116 elapsed -- Testing started completed.

[16:22:55]: ▸ 2024-12-17 16:22:55.779 xcodebuild[37567:51715957] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start

[16:22:55]: ▸ 2024-12-17 16:22:55.779 xcodebuild[37567:51715957] [MT] IDETestOperationsObserverDebug: 39.116 sec, +39.116 sec -- end

[16:22:57]: ▸ Failing tests:

[16:22:57]: ▸ 	ParametrizedFailsTests.testSampleFail()

[16:22:57]: ▸ 	-[ParametrizedFailsTests testParametrizedFails(:)]

[16:22:57]: ▸ 	-[ParametrizedFailsTests testParametrizedFails(:)]

[16:22:57]: ▸ 	-[ParametrizedFailsTests testParametrizedFails(:)]

[16:22:57]: ▸ ** TEST FAILED **

2024-12-17 16:22:12.887 xcodebuild[37567:51715969]  DVTDeviceOperation: Encountered a build number "" that is incompatible with DVTBuildVersion.

2024-12-17 16:22:12.890 xcodebuild[37567:51715957] [MT] DVTDeviceOperation: Encountered a build number "" that is incompatible with DVTBuildVersion.

--- xcodebuild: WARNING: Using the first of multiple matching destinations:

{ platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

{ platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

▸ Processing empty-ParametrizedFails.plist

▸ Linking ParametrizedFails.debug.dylib

▸ Linking ParametrizedFails

▸ Build Succeeded

--- xcodebuild: WARNING: Using the first of multiple matching destinations:

{ platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

{ platform:iOS Simulator, id:9F40E622-7C9C-4210-81B4-C59E0F02B7D7, OS:18.1, name:iPhone 16 Pro }

▸ Processing empty-ParametrizedFails.plist

▸ Linking ParametrizedFails.debug.dylib

▸ Linking ParametrizedFails

▸ Processing empty-ParametrizedFailsTests.plist

▸ Linking ParametrizedFailsTests

▸ Touching ParametrizedFailsTests.xctest (in target 'ParametrizedFailsTests' from project 'ParametrizedFails')

2024-12-17 16:22:55.779 xcodebuild[37567:51715957] [MT] IDETestOperationsObserverDebug: 39.116 elapsed -- Testing started completed.

2024-12-17 16:22:55.779 xcodebuild[37567:51715957] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start

2024-12-17 16:22:55.779 xcodebuild[37567:51715957] [MT] IDETestOperationsObserverDebug: 39.116 sec, +39.116 sec -- end

Failing tests:

ParametrizedFailsTests.testSampleFail()

-[ParametrizedFailsTests testParametrizedFails(:)]

-[ParametrizedFailsTests testParametrizedFails(:)]

-[ParametrizedFailsTests testParametrizedFails(:)]


** TEST FAILED **

[16:23:00]: Exit status: 65

+----------------------------------------+

|              Test Results              |

+--------------------+-------------------+

| Number of tests    | 2 (and 4 retries) |

| Number of failures | 1                 |

+--------------------+-------------------+


[16:23:00]: Tests have failed

[16:23:00]: Test execution failed. Exit status: 65

[16:23:01]: ---------------------

[16:23:01]: --- Step: trainer ---

[16:23:01]: ---------------------


+--------------------------------------------------------------------------------------------------------------------------+

|                                               Summary for trainer 2.226.0                                                |

+------------------------------+-------------------------------------------------------------------------------------------+

| path                         | /Users/evgeny/Library/Developer/Xcode/DerivedData/ParametrizedFails-gumghbybnfcsafffdduy  |

|                              | zfglmpwv/Logs/Test/Run-ParametrizedFails-2024.12.17_16-22-12-+0300.xcresult               |

| output_directory             | .                                                                                         |

| extension                    | .xml                                                                                      |

| fail_build                   | true                                                                                      |

| xcpretty_naming              | false                                                                                     |

| silent                       | false                                                                                     |

| output_remove_retry_attempts | false                                                                                     |

+------------------------------+-------------------------------------------------------------------------------------------+


[16:23:01]: Successfully generated './Run-ParametrizedFails-2024.12.17_16-22-12-+0300.xml'

+--------------------------------------------------------------------------------------------------------------------------+

|                                                       Lane Context                                                       |

+------------------------------+-------------------------------------------------------------------------------------------+

| DEFAULT_PLATFORM             | ios                                                                                       |

| PLATFORM_NAME                | ios                                                                                       |

| LANE_NAME                    | ios test                                                                                  |

| SCAN_GENERATED_XCRESULT_PATH | /Users/evgeny/Library/Developer/Xcode/DerivedData/ParametrizedFails-gumghbybnfcsafffdduy  |

|                              | zfglmpwv/Logs/Test/Run-ParametrizedFails-2024.12.17_16-22-12-+0300.xcresult               |

| SCAN_DERIVED_DATA_PATH       | /Users/evgeny/Library/Developer/Xcode/DerivedData/ParametrizedFails-gumghbybnfcsafffdduy  |

|                              | zfglmpwv                                                                                  |

| SCAN_GENERATED_PLIST_FILES   | []                                                                                        |

| SCAN_GENERATED_PLIST_FILE    |                                                                                           |

+------------------------------+-------------------------------------------------------------------------------------------+

[16:23:01]: Called from Fastfile at line 15

[16:23:01]: [16:23:01]:     13:	      fail_build: false, [16:23:01]:     14:	    ) [16:23:01]:  => 15:	    trainer( [16:23:01]:     16:	      path: lane_context[SharedValues::SCAN_GENERATED_XCRESULT_PATH], [16:23:01]:     17:	      output_directory: "." [16:23:01]:

[16:23:01]: Unit tests failed


+---------------------------------------+

|           fastlane summary            |

+------+------------------+-------------+

| Step | Action           | Time (in s) |

+------+------------------+-------------+

| 1    | default_platform | 0           |

| 2    | run_tests        | 52          |

| 💥   | trainer          | 0           |

+------+------------------+-------------+


[16:23:01]: fastlane finished with errors


[!] Unit tests failed

Contents of xml test report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="2" failures="1">
    <testsuite name="ParametrizedFailsTests" tests="2" failures="1" skipped="0" time="0.012132406234741211">
        <properties>
          <property name="Configuration" value="Test Scheme Action"/>
        </properties>
        <testcase classname="ParametrizedFailsTests" name="testSampleFail()" time="0.003058910369873047">
            <failure message="Expectation failed: (number → 5) &lt; 3
Number 5 is too big (/Users/evgeny/Documents/test/1/ParametrizedFails/ParametrizedFailsTests/ParametrizedFailsTests.swift#EndingLineNumber=13&amp;StartingLineNumber=13)">
            </failure>
        </testcase>
        <testcase classname="ParametrizedFailsTests" name="testParametrizedFails(_:)" time="6.604194641113281e-05">
        </testcase>
        <testcase classname="ParametrizedFailsTests" name="testParametrizedFails(_:)" time="0.0029311180114746094">
        </testcase>
        <testcase classname="ParametrizedFailsTests" name="testParametrizedFails(_:)" time="0.0029740333557128906">
        </testcase>
        <testcase classname="ParametrizedFailsTests" name="testParametrizedFails(_:)" time="0.0001461505889892578">
        </testcase>
        <testcase classname="ParametrizedFailsTests" name="testParametrizedFails(_:)" time="0.0029561519622802734">
        </testcase>
    </testsuite>
</testsuites>

We see that all testParametrizedFails(_:) looks like completed without failures, but in Xcode there will be three failed cases.

Environment

✅ fastlane environment ✅

Stack

Key Value
OS 15.1.1
Ruby 3.3.3
Bundler? true
Git git version 2.39.5 (Apple Git-154)
Installation Source /opt/homebrew/lib/ruby/gems/3.3.0/bin/fastlane
Host macOS 15.1.1 (24B91)
Ruby Lib Dir /opt/homebrew/Cellar/ruby/3.3.3/lib
OpenSSL Version OpenSSL 3.3.1 4 Jun 2024
Is contained false
Is homebrew false
Is installed via Fabric.app false
Xcode Path /Applications/Xcode.app/Contents/Developer/
Xcode Version 16.1
Swift Version 6.0.2

System Locale

Variable Value
LANG ru_RU.UTF-8 ✅
LC_ALL
LANGUAGE

fastlane files:

`./fastlane/Fastfile` 
default_platform(:ios)

platform :ios do

  desc "Run tests"
  lane :test do
    run_tests(
      scheme: "ParametrizedFails",
      device: "iPhone 16 Pro",
      # suppress_xcode_output: true,
      # skip_build: true,
      fail_build: false,
    )
    trainer(
      path: lane_context[SharedValues::SCAN_GENERATED_XCRESULT_PATH],
      output_directory: "."
    )
  end

end

No Appfile found

fastlane gems

Gem Version Update-Status
fastlane 2.226.0 ✅ Up-To-Date

Loaded fastlane plugins:

No plugins Loaded

Loaded gems
Gem Version
error_highlight 0.6.0
did_you_mean 1.6.3
syntax_suggest 2.0.0
bundler 2.5.11
pathname 0.3.0
rake 13.2.1
base64 0.2.0
nkf 0.2.0
rexml 3.4.0
CFPropertyList 3.0.7
public_suffix 6.0.1
addressable 2.8.7
artifactory 3.0.17
atomos 0.1.3
aws-eventstream 1.3.0
aws-partitions 1.1023.0
aws-sigv4 1.10.1
jmespath 1.6.2
aws-sdk-core 3.214.0
aws-sdk-kms 1.96.0
aws-sdk-s3 1.176.1
babosa 1.0.4
claide 1.1.0
colored 1.2
colored2 3.1.2
highline 2.0.3
commander 4.6.0
declarative 0.0.20
digest-crc 0.6.5
domain_name 0.6.20240107
dotenv 2.8.1
emoji_regex 3.2.3
excon 0.112.0
faraday-em_http 1.0.0
faraday-em_synchrony 1.0.0
faraday-excon 1.1.0
faraday-httpclient 1.0.1
multipart-post 2.4.1
faraday-multipart 1.0.4
faraday-net_http 1.0.2
faraday-net_http_persistent 1.2.0
faraday-patron 1.0.0
faraday-rack 1.0.0
faraday-retry 1.0.3
ruby2_keywords 0.0.5
faraday 1.10.4
http-cookie 1.0.8
faraday-cookie_jar 0.0.7
faraday_middleware 1.2.1
fastimage 2.3.1
sysrandom 1.0.5
fastlane-sirp 1.0.0
gh_inspector 1.1.3
jwt 2.9.3
multi_json 1.15.0
os 1.1.4
signet 0.19.0
googleauth 1.8.1
httpclient 2.8.3
mini_mime 1.1.5
trailblazer-option 0.1.2
uber 0.1.0
representable 3.2.0
retriable 3.1.2
google-apis-core 0.11.3
google-apis-androidpublisher_v3 0.54.0
google-apis-playcustomapp_v1 0.13.0
google-cloud-env 1.6.0
google-apis-iamcredentials_v1 0.17.0
google-apis-storage_v1 0.31.0
google-cloud-errors 1.4.0
google-cloud-core 1.7.1
google-cloud-storage 1.47.0
json 2.9.0
mini_magick 4.13.2
naturally 2.2.1
optparse 0.6.0
plist 3.7.1
rubyzip 2.3.2
security 0.1.5
simctl 1.6.10
terminal-notifier 2.0.0
unicode-display_width 2.6.0
terminal-table 3.0.2
tty-screen 0.8.2
tty-cursor 0.7.1
tty-spinner 0.9.3
word_wrap 1.0.0
nanaimo 0.4.0
xcodeproj 1.27.0
rouge 3.28.0
xcpretty 0.4.0
xcpretty-travis-formatter 1.0.1

generated on: 2024-12-17

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions