Skip to content

Scenario App Integration Tests will still pass even if some tests fail. #114805

@ColdPaleLight

Description

@ColdPaleLight

The following example is what I came across while developing PR flutter/engine#37342

You can see that the Scenario App Integration Tests passed here.
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Android%20Emulator%20Tests/3233/overview

But click on the details, and you can see that the internal test reports an error.
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8798157867833550513/+/u/Scenario_App_Integration_Tests/stdout?format=raw

[stdout] java.lang.AssertionError: expected:<0> but was:<-16776961>
[stdout] 	at org.junit.Assert.fail(Assert.java:88)
[stdout] 	at org.junit.Assert.failNotEquals(Assert.java:834)
[stdout] 	at org.junit.Assert.assertEquals(Assert.java:645)
[stdout] 	at org.junit.Assert.assertEquals(Assert.java:631)
[stdout] 	at dev.flutter.scenariosui.GetBitmapTests.getBitmap(GetBitmapTests.java:36)
[stdout] 	at java.lang.reflect.Method.invoke(Native Method)
[stdout] 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[stdout] 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[stdout] 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[stdout] 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[stdout] 	at androidx.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:531)
[stdout] 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
[stdout] 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
[stdout] 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
[stdout] 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
[stdout] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[stdout] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[stdout] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[stdout] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[stdout] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[stdout] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
[stdout] 	at androidx.test.runner.AndroidJUnit4.run(AndroidJUnit4.java:104)
[stdout] 	at org.junit.runners.Suite.runChild(Suite.java:128)
[stdout] 	at org.junit.runners.Suite.runChild(Suite.java:27)
[stdout] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[stdout] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[stdout] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[stdout] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[stdout] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[stdout] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
[stdout] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
[stdout] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
[stdout] 	at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
[stdout] 	at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
[stdout] 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2248)

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work listengineflutter/engine related. See also e: labels.platform-androidAndroid applications specificallyteam-androidOwned by Android platform teamtriaged-androidTriaged by Android platform team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions