This looks like a small breaking change for consumers of the tokenizer, since consecutive whitespaces are now collapsed into composite tokens.

Scalafix rules are impacted by that change since tokens are exposed at the document level and via each element of the tree without any indirection, so instead of trying to abstract that change away, I am considering signalling it via a breaking change bump.

Which got me wondering:

• In the interest of minimizing impact on Scalafix rule authors, are there at this stage any other upcoming breaking changes on the horizon?
• If not, do you think there would be a way to preserve the old behavior for a bit longer on the 4.9.x line? Probably not, as the changelog mentions "Support CRLF, multiple-whitespace sequences" so I assume it's a feature, not a side effect 😃

This looks like a small breaking change for consumers of the tokenizer, since consecutive whitespaces are now collapsed into composite tokens.

Oh well, I didn't realize scalafix was looking at tokens as well.

• In the interest of minimizing impact on Scalafix rule authors, are there at this stage any other upcoming breaking changes on the horizon?

We usually try to avoid breaking changes... that is, if we know they are breaking. But no, no plans.

• If not, do you think there would be a way to preserve the old behavior for a bit longer on the 4.9.x line? Probably not, as the changelog mentions "Support CRLF, multiple-whitespace sequences" so I assume it's a feature, not a side effect 😃

@tgodzik Sometime ago, we were debating whether to add some parameters to ScalametaParser, perhaps this might be one case. I think we ended up thinking Dialect is enough, don't remember why.

I guess in this case a setting would make sense, though not sure how to sensibly add it. We can have a separate tokenized method maybe?

We can have a separate tokenized method maybe?

Any chance you could give me some pointers so that I look at that to unblock the bump on Scalafix side? I went through the tokenisation code paths, but I am not sure where to start.

I guess we could just add another method to ScalametaTokenizer which will take options. We can then just use that explicitely from scalafix? If needed we can expose it later via the extension methods, but I think it should not be necessary right now.

@bjaglin let me do it in the next day or two, and release, if this is blocking you. I'm generally in the middle of fixing some bugs that i found by attempting to use scalafmt on the dotty codebase, but can complete the fixes in the next patch.

does scalafix use the tokenizer directly? or through the parser?

does scalafix use the tokenizer directly? or through the parser?

Only through the parser.