Background

Most of our runners assume some separation between the process that orchestrates running all the tests, and the tests themselves. This is different Isolates in the VM case, and external processes like browsers in other cases.

We also support running a _test.dart file directly:

This support is hardcoded to the expanded reporter and does not take any arguments for things like test filters.

In all cases when we run tests through a runner (internally, flutter_test, build_test, and package:test runner) we "wrap" the users library with out own and forward to their main in some way where the details vary by platform.

Goal

We should make it easier to combine the concepts of running without an external runner doing the orchestration, along with allowing some customization beyond the defaults. The immediate need is to register and pick a custom reporter.

Plan

Add a function in package:test_core/backend.dart like Future<bool> runTests(FutureOr<void> Function() userMain, {ReporterDetails reporter}).

Then platforms that want to build a custom "runner" that is baked in to a single execution instead of having a separate orchestration would use it by generating a wrapping library like:

import 'package:test_core/backend.dart';
import 'package:my_custom_test/my_custom_test.dart' as custom;  // The reporter implementation.

import '$userTestFile' as test; // Generated portion of the file - imports the users _test.dart file

void main() {
  runTests(test.main, reporter: custom.reporter);
}

We'd start with ReporterDetails for now, and we can hopefully add other named arguments for other customizations that crop up. For instance we might support test filtering by tag with an additional named argument.