Skip to content

Can we discuss how to update test262? #817

@morfav

Description

@morfav

Dear maintainers,

I thought it may be a good idea to update test262 to its current point.

It's taken me a lot longer (many hours) than I thought it would. The following issues cropped up:

  1. test262 has introduced extra "helper" (harness) files which use syntax that rhino does not understand (3-4 such files). It's necessary to create a custom version of these that rhino would run (or use a previous version of these files if it's an existing one). Do you think it's viable that these custom versions will be maintained with potentially further divergence in future?
  2. There are a lot of new, and some changed tests. Some of these pass, while others fail. It's a VERY time-consuming task to update the test262.properties file. I added all the new tests from git history to be excluded, however there were also quite a few (order of 1000) that pass, so need to be located and removed from the exclusions.
  3. A couple of new tests seem to never complete. To overcome this, I added a new status of "hard" exclude, which prevents them from being run (since currently tests marked to be excluded with an exclamation mark are still run, which causes the tests to hang).

I feel like it may make this whole process easier if we change the approach to running these tests. Namely, keep a version-controlled list of tests that PASS (rather than what is currently done, which is keep a list of failing tests).
When fixing bugs, it would be possible to add to the list of passing tests.
When syncing with the test262 repo, we have a different mode which runs all the non-passing tests to see which ones pass and can be (automatically) included (we still need a hard exclude to prevent running the ones that never complete). This mode can also be triggered to search for tests that are passing following commits to rhino. One potential complication would be that currently we can include whole directories, so would need to see how this fits with the new approach.

Please let me know what you think about the above, and whether there is an easier way to maintain test262 in sync.

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions