-
-
Notifications
You must be signed in to change notification settings - Fork 4.8k
feat: output full actual location in rule tester if different #19904
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
✅ Deploy Preview for docs-eslint canceled.
|
I'm not sure if I would agree with the described motivation since I believe the expected values should be specified based on expectations for the test case rather than copied from the actual values, but I'm fine with this change as I think it can be helpful for debugging to output the full actual location when at least one part of it doesn't match the expected value. @eslint/eslint-team thoughts? |
Oh! I actually like this a lot. Agreed that the expected values should be specified based on expectations for the test case rather than copied from the actual values. But also, changes to a rule's report range are quite tedious to reflect in tests. I've had to repeatedly go through passes of changing first one property, then another, etc. This would help. |
I'm personally 👍 to this change. I usually use Node.js's native test runner for my custom packages, and it shows less verbose error messages than mocha. So, I think this change would help users who use less verbose test runners for their tests. In my opinion, it would be nice to slightly beautify and summarize the newly added error message. |
Any suggestions? Maybe: AssertionError [ERR_ASSERTION]: Error endLine should be 8 - Actual: { line: 7, column: 16, endLine: 7, endColumn: 20 } ? |
If the desire here is to treat the location as one atomic unit then I don't think this is the right solution. Here, we're still checking the location one field at a time and the only difference is outputting all four fields...but you're still only telling people that one field is incorrect. Including the other three fields in the message doesn't mean anything because they could all be correct, all be incorrect, or something in between. If we really want to make a change here, then it should be to treat the four fields as one unit that are validated together and the failed assertions are shown together. |
I agree with @nzakas — the correct approach should be to treat the location fields as a single unit and validate them together, showing the actual vs expected values. That said, I'm unsure whether this change would be considered breaking or not. |
Treating it as a single unit would require validating all or none fields. assert.equals(
{ line: message.line, column: message.column, endLine: message.endLine, endColumn: message.endColumn },
{ line: error.line, column: error.column, endLine: error.endLine, endColumn: error.endColumn }
) I'm fine with that, but that would probably be more breaking than the error message of the method used to validate errors/messages. |
@ST-DDT that's not a problem, we can always check to see which of the location properties is present and only check those properties. Changing the assertion message output isn't a breaking change. It would only be breaking if we ended up checking a value that wasn't previously checked, which is not what we're doing. |
Do we still show all actual location properties? |
Only those that are present. What we should do is create an object with the properties that are defined in the test case, then create an object with the expected values for each property, then run |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
I havent updated the example in the PR description yet, but I expect it to change a bit after reviews anyway.
Switched to the default assertion output. |
Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com>
Side question: What is the |
We use this label to mark PRs that are considered for payment to contributors. For example: https://github.com/eslint/tsc-meetings/blob/main/notes/2025/2025-06-12.md#contributor-pool-may-2025 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks! Leaving open for @nzakas to verify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks!
In the future, please be sure to open an issue when proposing features instead of immediately opening a pull request.
Prerequisites checklist
What is the purpose of this pull request? (put an "X" next to an item)
[ ] Documentation update
[ ] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[ ] Add something to the core
[x] Other, please explain: Output the full actual location in rule-tester
What changes did you make? (Give an overview)
This PR changes the rule-tester error output to contain the actual error location in addition to the expected location.
In most cases, if one location property is wrong, then the others likely are as well, so outputing them all at once helps with updating all information in a single test run. This is especially useful for new rule tests or making existing tests more strict.
Old
New
The actual format is in JSON to make it easier to copy and paste, but I can beautify it, if you like.
Maybe:
.replaceAll('"', '').replaceAll(':', ': ').replaceAll(',', ', ')
Is there anything you'd like reviewers to focus on?