Previously: #18045 (comment)
Related: #19931 (comment), #19831

Docgen uses a different JavaScript parser (Espree) than that which is used in the plugin build process (Babel). This hasn't been much of an issue thus far, but with especially new syntaxes, it can cause errors to be thrown when running npm run docs:build and with the corresponding pre-commit step.

Examples:

• ES2019 optional catch binding
  • Was intended to be used in URL: Conform to URL Living Standard definition of valid URL #19871
• ES2020 optional chaining
  • Was intended to be used in Block Editor: LinkControl: Prevent focus loss in edit mode toggle #19931

Originally, I thought it might just be a matter of keeping the espree dependency up to date, assuming that it's been updated for newer versions of the specification.

This is unfortunately not as simple as it might seem, since one of the breaking changes introduced in espree@5.0.0 is to remove the attachComment feature, which is pretty critical to how docgen works (or at least is currently implemented). Thus, upgrading to the latest version would require a pretty significant refactor to the code, one where we'd likely emulate how comment attachment worked previously, using the comment (and perhaps tokens) option(s) to infer where a comment occurs in relation to an exported member. I started down this path on Friday, but it seems significant enough to warrant whether it's worth the effort, considering if the alternative to Doctrine would bring with it a different parser implementation as well (e.g. TypeScript).

One thing I noted in the process of this is that we configure the ECMAScript version here:

This is particularly relevant for my previous comment where I mention being unable to use ES2019 features. It's pretty clear by this code why this wouldn't be expected to work 😄 I haven't tested yet, but it's possible that Espree versions prior to the breaking 5.0 version might have support for newer versions of ECMAScript, so that we could still use those language features without bumping to the latest Espree version.

Task: Allow new syntaxes to be parsed without error by docgen.

Proposed solution:

One of:

1. Replace Doctrine and Espree with TypeScript (docgen: remove doctrine #18045).
2. Or: Bump Espree to the latest version, including necessary refactoring to docgen to support removal of attachComment option.
3. Or: Optionally bump Espree to the last version including attachComment option, and check to see whether it can support ES2019 and ES2020 as the configured ecmaVersion option.

Follow-up tasks:

Revisit the "intended to be used" examples from above, to introduce the desired syntax.

• https://github.com/WordPress/gutenberg/pull/19871/files#r372503756
• Block Editor: LinkControl: Prevent focus loss in edit mode toggle #19931 (comment)