-
Notifications
You must be signed in to change notification settings - Fork 900
Description
Dear maintainers,
I thought it may be a good idea to update test262 to its current point.
It's taken me a lot longer (many hours) than I thought it would. The following issues cropped up:
- test262 has introduced extra "helper" (harness) files which use syntax that rhino does not understand (3-4 such files). It's necessary to create a custom version of these that rhino would run (or use a previous version of these files if it's an existing one). Do you think it's viable that these custom versions will be maintained with potentially further divergence in future?
- There are a lot of new, and some changed tests. Some of these pass, while others fail. It's a VERY time-consuming task to update the test262.properties file. I added all the new tests from git history to be excluded, however there were also quite a few (order of 1000) that pass, so need to be located and removed from the exclusions.
- A couple of new tests seem to never complete. To overcome this, I added a new status of "hard" exclude, which prevents them from being run (since currently tests marked to be excluded with an exclamation mark are still run, which causes the tests to hang).
I feel like it may make this whole process easier if we change the approach to running these tests. Namely, keep a version-controlled list of tests that PASS (rather than what is currently done, which is keep a list of failing tests).
When fixing bugs, it would be possible to add to the list of passing tests.
When syncing with the test262 repo, we have a different mode which runs all the non-passing tests to see which ones pass and can be (automatically) included (we still need a hard exclude to prevent running the ones that never complete). This mode can also be triggered to search for tests that are passing following commits to rhino. One potential complication would be that currently we can include whole directories, so would need to see how this fits with the new approach.
Please let me know what you think about the above, and whether there is an easier way to maintain test262 in sync.
Thanks!