Skip to content

Conversation

eliasyishak
Copy link
Contributor

@eliasyishak eliasyishak commented Nov 11, 2022

Removing the default values from argParser for reporter and timeout. reporter doesn't need the default value compact passed to it at the flutter test level because that is already the default behavior for dart test.

Having a default value at this level overrides what is in dart_test.yaml because cli args are given priority over configured args.

For the same reason above, timeout default value has also been removed because dart test already defaults the timeout value to 30s

Fixes: #113507

Notes

There is a variable that test_core uses called reporter which has a defaultReporter backup if one is not provided within the Configuration class:

  /// The name of the reporter to use to display results.
  String get reporter => _reporter ?? defaultReporter;
  final String? _reporter;

https://github.com/dart-lang/test/blob/master/pkgs/test_core/lib/src/runner/configuration.dart#L94-L96

And when we don't pass an option for the reporter, you can see the logic for how defaultReporter is determined here:

final defaultReporter = inTestTests
    ? 'expanded'
    : inGithubContext
        ? 'github'
        : canUseSpecialChars
            ? 'compact'
            : 'expanded';

https://github.com/dart-lang/test/blob/master/pkgs/test_core/lib/src/runner/configuration/reporters.dart#L60-L67

This will depend on the canUseSpecialChars boolean which is defined by this code snippet below, and I believe this boolean resolves differently on windows which results in the error

/// Whether or not the current terminal supports ansi escape codes.
///
/// Otherwise only printable ASCII characters should be used.
bool get canUseSpecialChars =>
    (!Platform.isWindows || stdout.supportsAnsiEscapes) && !inTestTests;
https://github.com/dart-lang/test/blob/master/pkgs/test_core/lib/src/util/io.dart#L84-L88

https://github.com/dart-lang/test/blob/master/pkgs/test_core/lib/src/util/io.dart#L84-L88

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@flutter-dashboard flutter-dashboard bot added the tool Affects the "flutter" command-line tool. See also t: labels. label Nov 11, 2022
@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@eliasyishak eliasyishak marked this pull request as draft November 11, 2022 18:02
@flutter-dashboard
Copy link

This pull request has been changed to a draft. The currently pending flutter-gold status will not be able to resolve until a new commit is pushed or the change is marked ready for review again.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@eliasyishak
Copy link
Contributor Author

eliasyishak commented Nov 15, 2022

@Hixie this PR is looking to remove the default values for timeout when invoking flutter test – I noticed you left this annotation saying timeout isn't supported here though: https://github.com/eliasyishak/flutter/blob/test-yaml-ignored-113507/packages/flutter_tools/test/src/context.dart#L166-L168, is there a reason for this?

Or is this is only relevant for tests we run within flutter since it's within our context? Which shouldn't prevent use from removing the default values assigned by the arg parser

@Hixie
Copy link
Contributor

Hixie commented Nov 15, 2022

This change seems reasonable, thanks for making it.

@Hixie this PR is looking to remove the default values for timeout when invoking flutter test – I noticed you left this annotation saying timeout isn't supported here though: https://github.com/eliasyishak/flutter/blob/test-yaml-ignored-113507/packages/flutter_tools/test/src/context.dart#L166-L168, is there a reason for this?

That comment is just about testUsingContext, which is a flutter_tools internal test harness thing and shouldn't affect developers using Flutter.

The context is:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#never-check-if-a-port-is-available-before-using-it-never-add-timeouts-and-other-race-conditions
Basically our style guide bans the use of timeouts entirely. They are a constant source of flakes. At this point the only timeouts we should have in our own testing is to catch hangs, and those should be absurdly conservative and never ever be even remotely close to being hit by a functioning test. Specifically, there are two allowed timeouts: one on a per-shard basis at the CI level (about an hour per shard generally), and one on a per-test basis for shards with multiple tests, to enable the test to report more information about exactly where things hung before the per-shard timeout kills everything indiscriminately and without any useful debug output.

That's all somewhat off topic for this PR though, just giving context.

@eliasyishak eliasyishak marked this pull request as ready for review November 15, 2022 19:35
@@ -102,8 +102,10 @@ class _FlutterTestRunnerImpl implements FlutterTestRunner {
'--pause-after-load',
if (machine)
...<String>['-r', 'json']
else
...<String>['-r', reporter ?? 'compact'],
else if (globals.platform.environment['GITHUB_ACTIONS']?.toLowerCase() == 'true')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test framework defaults to github actions on github, you shouldn't need to set this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to move your logic for assigning github as the reporter downstream from where you put it.

https://github.com/flutter/flutter/pull/115137/files#diff-328d8a42e42b84152d577569aa043fdfa16977190165f1a11897f6f47d81c1a5

Is this not relevant?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you're removing the default, the test package's default logic for selecting the reporter should work, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I believe so. I found this snippet here within the repo for package:test_core which seems to be doing the same thing.

/// Detect whether we're running in a Github Actions context.
///
/// See
/// https://docs.github.com/en/actions/learn-github-actions/environment-variables.
bool get inGithubContext => Platform.environment['GITHUB_ACTIONS'] == 'true';

Seems like it gets used in the defaultReporter variable

final defaultReporter = inTestTests
    ? 'expanded'
    : inGithubContext
        ? 'github'
        : canUseSpecialChars
            ? 'compact'
            : 'expanded';

I'll go ahead and remove that conditional

Copy link
Contributor

@jonahwilliams jonahwilliams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eliasyishak
Copy link
Contributor Author

@christopherfujino fixing the tests to use the expanded results seems like much more effort, would it be better to capture that as an issue and revisit outside of the scope of this PR?

@eliasyishak eliasyishak added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 16, 2022
Copy link
Contributor

@christopherfujino christopherfujino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@auto-submit auto-submit bot merged commit d7454d5 into flutter:master Nov 16, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Nov 16, 2022
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Nov 16, 2022
* 338841a Revert "Revert "Revert "Scribble mixin (#104128)" (#114647)" (#114698)" (flutter/flutter#115146)

* 229b39e [flutter_tools] Fix so that the value set by `--dart-define-from-file` can be passed to Gradle (flutter/flutter#114297)

* 3895786 Revert "Load assets in flutter_test without turning event loop. (#115123)" (flutter/flutter#115156)

* a6e3ec8 mark firebase test lab test flaky because of upstream infra issues (flutter/flutter#115178)

* 8d808b5 Roll Flutter Engine from 7326e5b3354b to 329900fc85d6 (7 revisions) (flutter/flutter#115185)

* 10dc566 Add drone_dimensions to devicelab_build_test. (flutter/flutter#115183)

* 0a5b531 b6e45ae35 extend const_finder to allow skipping particular classes (flutter/engine#37257) (flutter/flutter#115189)

* 0e9ee36 [web] Add `--local-web-sdk` flag and use precompiled platform kernels for dart2js and ddc (flutter/flutter#114639)

* 9e314ff Make FutureBuilder handle SynchronousFuture correctly, reland SynchronousFuture usage in test assets (flutter/flutter#115173)

* 673fd97 17d428e68 Roll Dart SDK from 2b6909ba35e5 to 190040da0bc8 (2 revisions) (flutter/engine#37536) (flutter/flutter#115191)

* edcdb00 Roll Flutter Engine from 17d428e688a9 to 38c4f1d78630 (2 revisions) (flutter/flutter#115192)

* 8772768 Roll Flutter Engine from 38c4f1d78630 to 1f0dd2c0f5bb (2 revisions) (flutter/flutter#115194)

* 6ec2bd0 M3 Segmented Button widget (flutter/flutter#113723)

* b7b6758 f3dfdf4cc Roll Skia from 3bbdd5bc54f2 to a727f7ac8a98 (1 revision) (flutter/engine#37544) (flutter/flutter#115196)

* 94e753d Roll Flutter Engine from f3dfdf4ccf38 to e3b3950f0fab (3 revisions) (flutter/flutter#115200)

* 61e927d e63c9443b [Impeller] Include the new primitive type in the pipeline hash (flutter/engine#37546) (flutter/flutter#115201)

* c05d887 Roll Flutter Engine from e63c9443bc04 to 9500d8c4a4ce (2 revisions) (flutter/flutter#115212)

* 1f82985 1c8c23754 Roll Fuchsia Linux SDK from xDY5B0-NZzq2idRNU... to IE4logHFoa8ZZQAw3... (flutter/engine#37554) (flutter/flutter#115214)

* 61b3d5a a0d8cd68c Add a limit to the Gaussian blur downsampling curve (flutter/engine#37550) (flutter/flutter#115217)

* 2bf64aa fd56b7ee8 Roll Fuchsia Mac SDK from j3vspnGJcQvIfsXXu... to iPqmEVwAbmYwPlh5a... (flutter/engine#37557) (flutter/flutter#115218)

* 53ab17d Roll Flutter Engine from fd56b7ee87f0 to 1ae285897eae (2 revisions) (flutter/flutter#115220)

* 4488166 d4e2188e9 [Impeller] Fix DrawPaint regression (flutter/engine#37561) (flutter/flutter#115222)

* bf906e6 2bdeaf7d7 Fix inertia cancel event on macOS Ventura (flutter/engine#37067) (flutter/flutter#115226)

* 1a19409 06c390470 Roll Fuchsia Mac SDK from iPqmEVwAbmYwPlh5a... to N-iOLgSVYYh_AfC7Q... (flutter/engine#37566) (flutter/flutter#115236)

* 1a460cf 54795869b Roll Skia from c0f87aed019b to 38edf6197aa9 (1 revision) (flutter/engine#37567) (flutter/flutter#115239)

* 8230ef5 Revert "[web] Add `--local-web-sdk` flag and use precompiled platform kernels for dart2js and ddc (#114639)" (flutter/flutter#115242)

* 8994920 7269f2f91 Roll Skia from 38edf6197aa9 to 0f515e97d535 (1 revision) (flutter/engine#37571) (flutter/flutter#115244)

* 7ac00f7 40b071c75 Roll Fuchsia Linux SDK from IE4logHFoa8ZZQAw3... to AE_mBWzdDqUvn4r1N... (flutter/engine#37572) (flutter/flutter#115247)

* fa94a3c b63e6cf55 [Impeller] dont include sampler offset in float offset (flutter/engine#37573) (flutter/flutter#115248)

* cb224d1 b7567b92c Roll Fuchsia Mac SDK from N-iOLgSVYYh_AfC7Q... to WYzUv6RnWjJpAiqYp... (flutter/engine#37575) (flutter/flutter#115252)

* d587fe4 a1b5507e5 Add playground to demonstrate mask blur problems (flutter/engine#37574) (flutter/flutter#115260)

* f5205b1 6f7447efa Roll Fuchsia Linux SDK from AE_mBWzdDqUvn4r1N... to dRHIZSishiboEHMdw... (flutter/engine#37577) (flutter/flutter#115266)

* 2c4051f b077c1c46 Roll Dart SDK from 996fe4d7ea9a to 45b7433f9015 (1 revision) (flutter/engine#37578) (flutter/flutter#115271)

* 0b5f3fd 743cec5ba Roll Skia from 0f515e97d535 to b2109e6bbbe7 (1 revision) (flutter/engine#37580) (flutter/flutter#115279)

* 80bcc3a 035313ae2 Roll Dart SDK from 45b7433f9015 to 7cbcf48157cf (1 revision) (flutter/engine#37582) (flutter/flutter#115281)

* fdd2fe4 036f8ea88 Roll Fuchsia Mac SDK from WYzUv6RnWjJpAiqYp... to XvXKIR_SSUfSyGwxe... (flutter/engine#37583) (flutter/flutter#115293)

* 436fb4c c290de693 Roll Skia from b2109e6bbbe7 to 2ad14e25f6f7 (7 revisions) (flutter/engine#37585) (flutter/flutter#115294)

* 2449907 Fixed the color curve issue (flutter/flutter#115188)

* 9359c25 1a79e8686 [macOS] Move to new update semantics embedder API (flutter/engine#37404) (flutter/flutter#115301)

* 59e1edc Roll Flutter Engine from 1a79e8686036 to 8c73ffff5b6f (2 revisions) (flutter/flutter#115302)

* f2ec1c4 42bc762b6 Roll Skia from 2ad14e25f6f7 to a434f9b69660 (11 revisions) (flutter/engine#37591) (flutter/flutter#115304)

* ea4e11d Fix test in preparation of the Dart VM dropping support for language versions < 2.12.0 (flutter/flutter#115176)

* a0a7b3a 14ecaeb91 [Impeller] error earlier on uint types (flutter/engine#37593) (flutter/flutter#115308)

* 2e51077 Do not strip architecture suffixes from host local-engine (flutter/flutter#115320)

* 8e3ea14 Incorrect rendering of `SnapshotWidget` (flutter/flutter#114400)

* 59ecc75 [flutter_tools] add uint compilation test (flutter/flutter#115317)

* 1aeb172 Roll Flutter Engine from 14ecaeb915e0 to 78ae2414003a (10 revisions) (flutter/flutter#115335)

* e391812 9dedab305 Roll Skia from b474a43dcc34 to 2cebc1ce363c (2 revisions) (flutter/engine#37612) (flutter/flutter#115338)

* 9d64a0f 31ee28684 Combine results of all the test batches. (flutter/engine#37610) (flutter/flutter#115340)

* e9c6212 [Impeller] Add shader include with FlutterFragCoord for use by FragmentProgram (flutter/flutter#114214)

* a27bb3c 06bd5a816 Roll Skia from 2cebc1ce363c to d758b240cad0 (1 revision) (flutter/engine#37614) (flutter/flutter#115341)

* abbffcc c6a513ecd Roll Fuchsia Mac SDK from XvXKIR_SSUfSyGwxe... to 32kfSm94FGtJJeUjg... (flutter/engine#37615) (flutter/flutter#115344)

* 0f56ed1 f01de9f6b Roll Skia from d758b240cad0 to 24523449b1b3 (1 revision) (flutter/engine#37616) (flutter/flutter#115346)

* 25d74e6 0b338ed86 Roll Dart SDK from 6f5478a58387 to 987faaf6d1d4 (2 revisions) (flutter/engine#37617) (flutter/flutter#115352)

* 1f28f99 721fbe111 Roll Skia from 24523449b1b3 to 4d519a8ba97a (3 revisions) (flutter/engine#37618) (flutter/flutter#115354)

* c37f255 [tools] Fix plugin_ffi template lint violation (flutter/flutter#115356)

* 3c3f136 b4fd07fa4 Roll Dart SDK from 987faaf6d1d4 to 42f87c0521f8 (1 revision) (flutter/engine#37619) (flutter/flutter#115362)

* 2ed5283 58495f077 [Impeller] Add debug names to additional VK objects (flutter/engine#37592) (flutter/flutter#115363)

* afafde6 7f74f34a4 Roll Fuchsia Linux SDK from B0OuUvWOY24LI1WoF... to agStiOtzmc9Fmw6gc... (flutter/engine#37621) (flutter/flutter#115365)

* af2ec91 Roll Plugins from 3ca3410 to e500884 (10 revisions) (flutter/flutter#115366)

* 3a298d0 Roll Flutter Engine from 7f74f34a408a to eead7b0d8656 (2 revisions) (flutter/flutter#115369)

* acf01eb 905862722 [dart] Run generate_sdk_version_file.py to be compatible with dart ch… (flutter/engine#37624) (flutter/flutter#115372)

* 95ace11 Include initial offset when using PlatformViewSurface (flutter/flutter#114103)

* 39a9ed5 Updated tokens to v0.141 (flutter/flutter#115298)

* ba47c29 216702f3d Roll Skia from 1991515dd191 to 826b3ea577f3 (4 revisions) (flutter/engine#37628) (flutter/flutter#115377)

* 2d77ac5 49f8326df Roll Skia from 826b3ea577f3 to 02e2101a75f7 (4 revisions) (flutter/engine#37631) (flutter/flutter#115385)

* 136b46b Hint text semantics to be excluded in a11y read out if textfield in not empty and label text is provided  (flutter/flutter#115010)

* 700de09 Change button and label text to sentence case for Material 3 (flutter/flutter#115187)

* b2b8391 Roll Flutter Engine from 49f8326df6f0 to 84cb1f833ac4 (3 revisions) (flutter/flutter#115387)

* 2c44f4a Remove dev branch reference from build ios-frameworks error (flutter/flutter#115166)

* 792eefe Roll Flutter Engine from 84cb1f833ac4 to af6e47f1d5c4 (4 revisions) (flutter/flutter#115395)

* 341a9b1 6d41b2ffb update Scene.toImageSync test to use proper bounds in the request (flutter/engine#37638) (flutter/flutter#115398)

* e66183d Bump github/codeql-action from 2.1.25 to 2.1.32 (flutter/flutter#115394)

* d0e5b2f Add a way to customize padding in BottomAppBar  (flutter/flutter#115175)

* 349d648 baad8f2c5 Roll Skia from 02e2101a75f7 to cfb6de45f39b (7 revisions) (flutter/engine#37640) (flutter/flutter#115399)

* 7d2b011 Fix logic error in `markNeedsPaint` (flutter/flutter#112735)

* 93c4b09 Add `RestorableEnumN<T>` and `RestorableEnum<T>` to restorable primitive types (flutter/flutter#115050)

* 2c1536a Create a main alias for master channel. (flutter/flutter#115388)

* bbb349b Roll Flutter Engine from baad8f2c553b to e56ed93faab5 (6 revisions) (flutter/flutter#115406)

* bacd770 c062f7ea7 Roll Skia from cfb6de45f39b to cdf8348e6fba (4 revisions) (flutter/engine#37649) (flutter/flutter#115407)

* 27ff6a5 Roll Flutter Engine from c062f7ea7195 to 03e9a97f8abe (3 revisions) (flutter/flutter#115410)

* 58728c6 Roll Flutter Engine from 03e9a97f8abe to 21a572e4e4d0 (2 revisions) (flutter/flutter#115411)

* e54c8a8e2 3e15c1925 Roll Dart SDK from 7163c9627d98 to c3f1b3642181 (2 revisions) (flutter/engine#37664) (flutter/flutter#115417)

* 484e09e Roll Flutter Engine from 3e15c192560f to 0241f18cbedf (2 revisions) (flutter/flutter#115423)

* 4fdaf7a Revert "Roll Flutter Engine from 3e15c192560f to 0241f18cbedf (2 revisions) (#115423)" (flutter/flutter#115456)

* 4301731 Make Flutter Driver actively wait for runnable isolate (flutter/flutter#113969)

* c940f31 Marks Mac_ios flavors_test_ios to be unflaky (flutter/flutter#115458)

* 243a830 Roll Flutter Engine from 3e15c192560f to 223e1f092dd5 (4 revisions) (flutter/flutter#115463)

* d7454d5 removing default values for [reporter] and [timeout] in flutter test (flutter/flutter#115160)

* a2233ea [flutter_tools] remove all body_might_complete_normally_catch_error ignores (flutter/flutter#115184)

* b5345ff d72164776 Revert "[Impeller] Refactor color source resolution to use explicit factory types (#37656)" (flutter/engine#37673) (flutter/flutter#115467)

* dcae424 [tools]build ipa validate template icon files (flutter/flutter#114841)
@eliasyishak eliasyishak deleted the test-yaml-ignored-113507 branch November 17, 2022 18:15
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 17, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Nov 18, 2022
shogohida pushed a commit to shogohida/flutter that referenced this pull request Dec 7, 2022
…lutter#115160)

* removing default values for [reporter] and [timeout]]

* passing reporter arg to see tests pass

* added test to confirm TestCommand is not passing defaults

* add'l helper message for [reporter] arg

* default behavior for github actions + fixed tests

* removing github conditional for reporter + related test

* removing unused import
gspencergoog pushed a commit to gspencergoog/flutter that referenced this pull request Jan 19, 2023
…lutter#115160)

* removing default values for [reporter] and [timeout]]

* passing reporter arg to see tests pass

* added test to confirm TestCommand is not passing defaults

* add'l helper message for [reporter] arg

* default behavior for github actions + fixed tests

* removing github conditional for reporter + related test

* removing unused import
auto-submit bot pushed a commit that referenced this pull request May 8, 2023
Close #125940

I will add tests if this PR looks roughly OK :)

The fix mainly mimics #115160 - just remove the default argument.

p.s. I ran into this bug when wanting to set concurrency in my dart_test.yaml for one set of my tests which I need to be executed without parallalization.
fluttermirroringbot pushed a commit that referenced this pull request May 8, 2023
Close #125940

I will add tests if this PR looks roughly OK :)

The fix mainly mimics #115160 - just remove the default argument.

p.s. I ran into this bug when wanting to set concurrency in my dart_test.yaml for one set of my tests which I need to be executed without parallalization.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autosubmit Merge PR when tree becomes green via auto submit App tool Affects the "flutter" command-line tool. See also t: labels.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Some values in dart_test.yaml are ignored when running flutter test
4 participants