Add capabilities for building direct test runners #1332
Conversation
Towards #1310 Open questions: - Can we make coverage work? How? - This currently prints full stack traces on failure. Is that OK? Do we need to install a top-level zone with an uncaught error handler to truncate them?
|
Maybe for coverage we could let the user wire that up themselves outside of this function? Another possible approach along the lines of #1310 (comment) could be to expose a |
I think we'll probably need to. @grouma do you have an idea of what type of API we'd want to write to make it easy to trigger coverage collection for the main isolate? This differs a little from our normal approach where we are interested in the coverage from an isolate we spawn and control.
I spent some more time thinking about this approach and I don't think it's really feasible. I'll update on the linked issue. |
I'm not sure if that is even feasible. To collect coverage you need to provide a service URI. I don't know how the test file would get access to the VM's service URI. I'm also not sure what happens when a VM starts inspecting itself. If a VM can't inspect itself, you may be able to do something super hacky like spawning a detached process to do the inspection. That seems incredibly weird to me though. Finally, I would expect the user would likely have to pass |
|
Wouldn't the test file be able to import |
I was not aware of this workflow. If this works with Flutter then it will almost certainly work with the vanilla VM. In regards to Nates original question then, the API should probably just mirror that of the |
Add `filterToSingleTestName` to `Declarer`. Any groups or test cases which do not match the filter will be excluded. This means that any `solo` use on tests that are excluded will not be honored. Add `enumeratTestCases` and `directRunSingleTest` APIs.
|
@jakemac53 @grouma - how do you feel about merging this without tests? I'll keep it in |
| @@ -195,6 +215,11 @@ class Declarer { | |||
| bool solo = false}) { | |||
| _checkNotBuilt('group'); | |||
|
|
|||
| final fullName = _prefix(name); | |||
| if (!(_filterToSingleTestName?.startsWith(fullName) ?? true)) { | |||
There was a problem hiding this comment.
Possibly worth a comment?
I also find the ?? true wrapped in parens and then negated very difficult to parse. I would write a more verbose form.
There was a problem hiding this comment.
I'll need to use ! for the verbose form since it's a field. That's why I wrote it this way originally. I can expand it with the ! though.
| /// test a [DuplicateTestNameException] will be thrown. | ||
| /// | ||
| /// Skipped tests are ignored. | ||
| Future<Iterable<String>> enumerateTestCases( |
There was a problem hiding this comment.
Should we type this as Set instead of Iterable - the contract is guaranteeing uniqueness so I think it makes sense and would avoid some potential copies for users who want a set.
There was a problem hiding this comment.
I'm uncertain when I want to used an ordered collection. As implemented the test names are returned in the same order that they'd be run (without randomization turned on). I don't know if that matters or not, or if it's better to return it as a Set. I'll try Set for now.
Also fix up changelog and docs
directRunTests
Towards #1310, #1328
Add
directRunTestto configure a reporter and run tests directly inthe same isolate.
Add
enumerateTestCasesto collect test names without running them, anddirectRunSingleTestto run a specific test by its full name. TheseAPIs ensure the uniqueness of test names to avoid ambiguity. This
restriction may be spread to tests run through the normal test runner as
well.
fullTestNameoption onDeclarer. When used, only the test (ortests if uniqueness is not checked separately) will be considered as a
test case.
directRunTest,enumerateTestCases, anddirectRunSingleTestAPIs. These are kept under a
lib/src/import for now, and any otherpackage that uses these APIs should pin to a specific version of
package:test_core. The details of these APIs might change without amajor version bump.