Flake reduction has been mainly focused on devicelab tests, while the shard tests (ones contain many unit/integration tests) are not being tracked actively. This bug tracks different flakes on different shard tests. Data is based Q4 2021 flakes happened from 10/01/2021 to 01/13/2022

GOAL: proactively fix/skip all flaky tests to achieve zero flake on these tests

build_tests

• Linux build_tests_1_2: 1 flake, fixed in Engine -> FW roller failing due to gradle resolution errors #96550
• Windows build_tests
  • Windows build_tests_1_3 & build_tests_2_3: 3 flakes on android_aot_release_android-arm64 failure
  • Windows build_tests_2_3: 2 flakes on build apk --release -v, Comparison method violates its general contract! error across many Android tests. #103230

tool_tests

• tool_tests_commands: 1 flake on packages_test TimeoutException, The tool_tests_commands flakes on test\commands.shard\permeable\packages_test.dart TimeoutException #97831

framework_tests

• Mac framework_tests_libraries: 1 flake on date_picker_test, framework_tests_libraries flakes` on date_picker_test #97161
• Linux framework_tests_misc: 1 flake on cupertino_action_sheet , Linux framework_tests_misc flakes on cupertino_action_sheet #100641
• Linux framework_tests_libraries: 1 flake on bottom_navigation_bar_test, Linux framework_tests_libraries flakes on bottom_navigation_bar_test #97162
• Linux framework_tests_widgets: 2 flakes: 1 dismissible_test flake, 1 aspect_ratio_test, fixed in Linux framework_tests_widgets is 2.00% flaky #94729
• Windows framework_tests_widgets: 4 flakes, Windows framework_tests_widgets flakes on timeout #97163
• Windows framework_tests_libraries:
  • 1 flake on date_picker_test, framework_tests_libraries flakes` on date_picker_test #97161
  • 2 flakes on platform_view_test, framework_tests_libraries flakes on platform_view_test #100249
• Windows framework_tests_misc: 1 flake on live_widget_controller_test flake, Windows framework_tests_misc flakes on live_widget_controller_test #97165

tool_integration_tests

• Windows tool_integration_tests_1_6:
  • 1 flakes on android_plugin_example_app_build_test, Windows tool_integration_tests_1_6 flakes on android_plugin_example_app_build_test #98085
  • 3 flakes on android_plugin_ndkversion_mismatch_test, Windows tool_integration_tests_1_6 flakes on android_plugin_ndkversion_mismatch_test #105813, Comparison method violates its general contract! error across many Android tests. #103230
• Windows tool_integration_tests_2_6 flakes on deferred_components_test, Windows tool_integration_tests_2_6 flakes on deferred_components_test #101609
• Windows tool_integration_tests_3_6:
  • 1 flakes on expression_evaluation_test, Windows tool_integration_tests_3_6 flakes on expression_evaluation_test #103215
  • 1 flakes on flutter_build_null_unsafe_test, Windows tool_integration_tests_3_6 flakes on flutter_build_null_unsafe_test #98140
  • 1 flake on flutter_build_with_compilation_error_test, Windows tool_integration_tests_3_6 flakes on flutter_build_with_compilation_error_test #101950
• Windows tool_integration_tests_1_5: 7 flakes on timeout, fixed Windows tool_integration_tests_1_5 is 3.00% flaky #95584
• Windows tool_integration_tests_2_5: 3 flakes: 1 timeout, 2 test flakes Windows tool_integration_tests_2_5 tests flake #97231
• Windows tool_integration_tests_3_5: 1 flake on 1 timeout, fixed Windows tool_integration_tests_1_5 is 3.00% flaky #95584
• Windows tool_integration_tests_4_5: 2 flakes, Windows tool_integration_tests_4_5 flake on overall_experience_test #97232
• Windows tool_integration_tests_5_5: 3 flakes on test, Windows tool_integration_tests_5_5 flakes on tests #97234
• Mac tool_integration_tests_3_4: 1 flake, Mac tool_integration_tests_3_4 flakes on flutter_adapter_test #97238
• Linux tool_integration_tests_1_4:
  • 1 flake on observatory_port_test timeout, Linux tool_integration_tests_1_4 flakes on observatory_port_test timeout #97240
  • 1 flake on gradle_non_android_plugin_test, Linux tool_integration_tests_1_4 flakes on gradle_non_android_plugin_test #99767
• Linux tool_integration_tests_3_4: 2 flakes: 2 test flakes, Linux tool_integration_tests_3_4 is 2.00% flaky #94030
• Linux tool_integration_tests_4_4:
  • 10 assertion flakes, Linux tool_integration_tests_4_4 failing assertion #92042,
  • 4 flakes on multiple unit tests, Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245
  • 1 flake on cache_test, Linux tool_integration_tests_4_4 flakes on cache_test #98141

web_tool_tests

• Linux web_tool_tests
  • 2 flakes on Unexpected character, Intermittent failure to connect to chrome tabs on test bots #94886
  • 4 flakes on Failed to create server socket, Linux web_tool_tests flakes on Failed to create server socket #97332

web_tests

• Linux web_tests flakes on Connection refused+timeout, Linux web_tests flakes on Connection refused+timeout #97324
• Linux web_tests_4
  • 1 flake on list_wheel_scroll_view_test, Linux web_tests_4 flakes on list_wheel_scroll_view_test #103217
  • flakes on Segmentation fault, Linux web_tests_4 is 2.00% flaky #99004
• Linux web_tests_6
  • 1 flake on runtime/vm/object.cc. Linux web_tests flakes_6 flakes on runtime/vm/object.cc error #97326
  • 1 flake on dialog_test, Linux web_tests flakes_6 flakes on test dialog_test #99942

web_long_running_tests

• Linux web_long_running_tests_1_5
  • 1 flake due to malformed response from the FlutterDriver extension, Linux web_long_running_tests flakes due to malformed response from the FlutterDriver extension #102385
  • 1 flake on version info not in JSON, Linux web_long_running_tests flakes with error version info not in JSON #97307, Linux web_long_running_tests_1_5 is 2.00% flaky #102660
  • 10 flakes on The getter 'uri' was called on null, Linux web_long_running_tests_1_5 is 2.00% flaky #94031, Linux tool_integration_tests_4_4 failing assertion #92042
  • 1 flake on timeout, Linux web_long_running_tests_1_5 flakes on timeout #97312
  • 1 flake on Chrome failed to start, Linux web_long_running_tests_1_5 flakes on Chrome failed to start #97314
  • 1 flake on image_loading_integration, Linux web_long_running_tests_1_5 flakes on image_loading_integration #97315
  • 1 flake on Unable to start WebDriver Session, Linux web_long_running_tests flakes on Unable to start WebDriver Session #97316
• Linux web_long_running_tests_2_5
  • 1 flake on timeout, Linux web_long_running_tests_2_5 flakes on timeout #97317
  • 1 flake on Unexpected character, Linux web_long_running_tests_2_5 flakes on Unexpected character #97318
  • 1 flake on version info not in JSON, Linux web_long_running_tests flakes with error version info not in JSON #97307
  • 1 flake on Failed to create server socket, Linux web_long_running_tests flakes on Failed to create server socket #97319
  • 1 flake on Unable to start WebDriver Session, Linux web_long_running_tests flakes on Unable to start WebDriver Session #97316
• Linux web_long_running_tests_3_5
  • 1 flake on timeout, Linux web_long_running_tests_3_5 flakes on timeout #97320
  • 1 flake on Unable to start WebDriver Session, Linux web_long_running_tests flakes on Unable to start WebDriver Session #97316
• Linux web_long_running_tests_4_5
  • 1 flake on Failed to create server socket, Linux web_long_running_tests flakes on Failed to create server socket #97319
  • 1 flake on Unable to start WebDriver Session, Linux web_long_running_tests flakes on Unable to start WebDriver Session #97316
• Linux web_long_running_tests_5_5
  • 1 flake on type 'Null' is not a subtype of type 'Map<String, dynamic>' in type cast, Linux web_long_running_tests_5_5 flakes on type 'Null' is not a subtype of type 'Map<String, dynamic>' in type cast #97322
  • 1 flake on Failed to create server socket, Linux web_long_running_tests flakes on Failed to create server socket #97319
  • 1 flake on Unable to start WebDriver Session, Linux web_long_running_tests flakes on Unable to start WebDriver Session #97316