Skip to content

Conversation

ST-DDT
Copy link
Contributor

@ST-DDT ST-DDT commented Jul 1, 2025

Prerequisites checklist

What is the purpose of this pull request? (put an "X" next to an item)

[ ] Documentation update
[ ] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[ ] Add something to the core
[x] Other, please explain: Output the full actual location in rule-tester

What changes did you make? (Give an overview)

This PR changes the rule-tester error output to contain the actual error location in addition to the expected location.
In most cases, if one location property is wrong, then the others likely are as well, so outputing them all at once helps with updating all information in a single test run. This is especially useful for new rule tests or making existing tests more strict.

Old

AssertionError [ERR_ASSERTION]: Error endLine should be 8

7 !== 8

+ expected - actual

-7
+8

New

AssertionError [ERR_ASSERTION]: Actual error location does not match expected error location.
+ actual - expected

{
+   column: 11,
+   endColumn: 16,
+   endLine: 1,
+   line: 1
-   column: 0,
-   endColumn: 0,
-   endLine: 0,
-   line: 0
}

+ expected - actual

{
-  "column": 11
-  "endColumn": 16
-  "endLine": 1
-  "line": 1
+  "column": 0
+  "endColumn": 0
+  "endLine": 0
+  "line": 0
}

The actual format is in JSON to make it easier to copy and paste, but I can beautify it, if you like.

Maybe: .replaceAll('"', '').replaceAll(':', ': ').replaceAll(',', ', ')

Is there anything you'd like reviewers to focus on?

@ST-DDT ST-DDT requested a review from a team as a code owner July 1, 2025 08:31
@github-project-automation github-project-automation bot moved this to Needs Triage in Triage Jul 1, 2025
@eslint-github-bot eslint-github-bot bot added the feature This change adds a new feature to ESLint label Jul 1, 2025
Copy link

netlify bot commented Jul 1, 2025

Deploy Preview for docs-eslint canceled.

Name Link
🔨 Latest commit 3c8e6a4
🔍 Latest deploy log https://app.netlify.com/projects/docs-eslint/deploys/6867f317c32e23000857bdce

@github-actions github-actions bot added the core Relates to ESLint's core APIs and features label Jul 1, 2025
@mdjermanovic mdjermanovic moved this from Needs Triage to Triaging in Triage Jul 1, 2025
@mdjermanovic
Copy link
Member

I'm not sure if I would agree with the described motivation since I believe the expected values should be specified based on expectations for the test case rather than copied from the actual values, but I'm fine with this change as I think it can be helpful for debugging to output the full actual location when at least one part of it doesn't match the expected value.

@eslint/eslint-team thoughts?

@mdjermanovic mdjermanovic moved this from Triaging to Feedback Needed in Triage Jul 1, 2025
@mdjermanovic mdjermanovic added the evaluating The team will evaluate this issue to decide whether it meets the criteria for inclusion label Jul 1, 2025
@JoshuaKGoldberg
Copy link
Contributor

Oh! I actually like this a lot. Agreed that the expected values should be specified based on expectations for the test case rather than copied from the actual values. But also, changes to a rule's report range are quite tedious to reflect in tests. I've had to repeatedly go through passes of changing first one property, then another, etc. This would help.

@lumirlumir
Copy link
Member

I'm personally 👍 to this change.

I usually use Node.js's native test runner for my custom packages, and it shows less verbose error messages than mocha.

So, I think this change would help users who use less verbose test runners for their tests.


In my opinion, it would be nice to slightly beautify and summarize the newly added error message.

@ST-DDT
Copy link
Contributor Author

ST-DDT commented Jul 1, 2025

In my opinion, it would be nice to slightly beautify and summarize the newly added error message.

Any suggestions?

Maybe:

AssertionError [ERR_ASSERTION]: Error endLine should be 8 - Actual: { line: 7, column: 16, endLine: 7, endColumn: 20 }

?

@nzakas
Copy link
Member

nzakas commented Jul 1, 2025

If the desire here is to treat the location as one atomic unit then I don't think this is the right solution. Here, we're still checking the location one field at a time and the only difference is outputting all four fields...but you're still only telling people that one field is incorrect. Including the other three fields in the message doesn't mean anything because they could all be correct, all be incorrect, or something in between.

If we really want to make a change here, then it should be to treat the four fields as one unit that are validated together and the failed assertions are shown together.

@snitin315
Copy link
Contributor

I agree with @nzakas — the correct approach should be to treat the location fields as a single unit and validate them together, showing the actual vs expected values. That said, I'm unsure whether this change would be considered breaking or not.

@ST-DDT
Copy link
Contributor Author

ST-DDT commented Jul 2, 2025

Treating it as a single unit would require validating all or none fields.

assert.equals(
  { line: message.line, column: message.column, endLine: message.endLine, endColumn: message.endColumn },
  { line: error.line, column: error.column, endLine: error.endLine, endColumn: error.endColumn }
)

I'm fine with that, but that would probably be more breaking than the error message of the method used to validate errors/messages.
I assume, nobody is checking the failure messages, only whether the rule tester passes or fails.
However, I think quite a few tests/devs only verify a few/subset of the location properties.

@nzakas
Copy link
Member

nzakas commented Jul 2, 2025

@ST-DDT that's not a problem, we can always check to see which of the location properties is present and only check those properties.

Changing the assertion message output isn't a breaking change. It would only be breaking if we ended up checking a value that wasn't previously checked, which is not what we're doing.

@ST-DDT
Copy link
Contributor Author

ST-DDT commented Jul 2, 2025

Do we still show all actual location properties?
Or only those that are already present?
I would prefer showing all.

@nzakas
Copy link
Member

nzakas commented Jul 2, 2025

Only those that are present. What we should do is create an object with the properties that are defined in the test case, then create an object with the expected values for each property, then run assert.deepStrictEqual() on those objects to get the desired output.

Copy link
Contributor Author

@ST-DDT ST-DDT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

I havent updated the example in the PR description yet, but I expect it to change a bit after reviews anyway.

@ST-DDT
Copy link
Contributor Author

ST-DDT commented Jul 3, 2025

Switched to the default assertion output.

@ST-DDT
Copy link
Contributor Author

ST-DDT commented Jul 4, 2025

Side question: What is the contributor pool? The label doesn't have a description.

@mdjermanovic
Copy link
Member

Side question: What is the contributor pool? The label doesn't have a description.

We use this label to mark PRs that are considered for payment to contributors. For example:

https://github.com/eslint/tsc-meetings/blob/main/notes/2025/2025-06-12.md#contributor-pool-may-2025

Copy link
Member

@mdjermanovic mdjermanovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! Leaving open for @nzakas to verify.

@mdjermanovic mdjermanovic moved this from Feedback Needed to Second Review Needed in Triage Jul 4, 2025
@mdjermanovic mdjermanovic added accepted There is consensus among the team that this change meets the criteria for inclusion and removed evaluating The team will evaluate this issue to decide whether it meets the criteria for inclusion labels Jul 4, 2025
Copy link
Member

@nzakas nzakas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

In the future, please be sure to open an issue when proposing features instead of immediately opening a pull request.

@nzakas nzakas merged commit 35cf44c into eslint:main Jul 7, 2025
30 checks passed
@github-project-automation github-project-automation bot moved this from Second Review Needed to Complete in Triage Jul 7, 2025
@ST-DDT ST-DDT deleted the rule-tester/full-location branch July 7, 2025 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted There is consensus among the team that this change meets the criteria for inclusion contributor pool core Relates to ESLint's core APIs and features feature This change adds a new feature to ESLint
Projects
Status: Complete
Development

Successfully merging this pull request may close these issues.

6 participants