[1.9.x] Smooth transition to valid Maven pattern of sbt plugins #7096

adpi2 · 2022-12-21T08:41:30Z

The goal of this PR is to fix #3410 in sbt 1.x without creating a big-bang in the ecosystem of sbt plugins.

Summary

The implemented strategy is:

sbt 1.9 publishes valid poms and old poms
Coursier 2.1 and sbt 1.9 try first to resolve the valid pom and fallback to the invalid pom if the valid pom is missing (backward compatibility)
Older versions of Coursier and sbt can still resolve newly published sbt plugins from its old pom (forward compatibility)

Thus the migration of plugins can happen in any order:

A plugin can migrate to sbt 1.9.x even if it depends on a plugin published by sbt <= 1.8.x.
A plugin can stay on sbt <= 1.8.x even if it depends on a plugin published by 1.9.x.

It is worth noting that the descriptor of an sbt plugin dependency is still the same: org.example:example:1.0.0 with extra-attributes scalaVersion:2.12 and sbtVersion:1.0

Motivation

From the user perspective almost nothing changes, the dependency to a plugin is still declared as:

addSbtPlugin("org.example" % "example" % "1.0.0")

The added value is:

resolution of an sbt plugin works in enterprise environments
mvnrepository.com starts indexing the sbt plugins
statistics (downloads) are available on Sonatype for sbt plugins
javadoc.io can find the scaladoc of sbt plugins
we can fix the linking of sbt plugin dependencies in Scaladex
we can use Maven to resolve sbt plugins
(we can use other tools that depends on Maven to resolve sbt plugins)

Associated PRs

In Coursier: Try resolve sbt plugin from valid Maven pattern coursier/coursier#2633 contains the resolution of an sbt plugin based on the valid pattern and the parsing of the valid pom
In librarymanagement: Try resolve sbt plugin from valid Maven pattern librarymanagement#409 contains the resolution of an sbt plugin based on the valid pattern, the parsing of the valid pom and the generation of the valid pom

In depth

Generating the valid pom file

We consider a pom file is valid if Maven can resolve it and check its consistency. More precisely, the path of the pom file and the declared artifactId in the pom file must be consistent.

Given module org.example:example:1.0.0 with extra-attributes scalaVersion:2.12 and sbtVersion:1.0, its valid Maven path must be org/example/example_2.12_1.0/1.0.0/example_2.12_1.0-1.0.0.pom and the corresponding Maven module ID is:

<groupId>org.example</groupId>
<artifactId>example_2.12_1.0</artifactId>
<version>1.0.0</version>
<properties>
  <scalaVersion>2.12</scalaVersion>
  <sbtVersion>1.0</sbtVersion>
</properties>

Maven must also be able to resolve the sbt plugin dependencies. Thus all declared sbt plugin dependencies in the pom file must contain valid artifact IDs, with the sbt cross-version.

This pom file is generated as if the declared module ID is org.example:example_2.12_1.0:1.0.0 with no extra attributes. However, in sbt and Coursier, we must maintain the old-style module ID, with extra-attributes, to get the bi-directional compatibilty.

Resolving an sbt plugin

Given module org.example:example:1.0.0 with extra-attributes scalaVersion:2.12 and sbtVersion:1.0, we first try to resolve org/example/example_2.12_1.0/1.0.0/example_2.12_1.0-1.0.0.pom and we fallback to org/example/example_2.12_1.0/1.0.0/example-1.0.0.pom.

Whenever we parse a pom file, we must make sure to remove the cross-version part if it is redundant with the extra-attributes.
So parsing example_2.12_1.0-1.0.0.pom returns the same module ID as parsing example-1.0.0.pom. That is org.example:example:1.0.0 with extra-attributes scalaVersion:2.12 and sbtVersion:1.0. All dependencies, in the pom file, are also treated the same. The parsed dependencies of example_2.12_1.0-1.0.0.pom are the same as the ones of example-1.0.0.pom. Thus the conflict resolution is working well: sbt can resolve conflicts between old and new poms, old and new dependencies.

Tests

To test these changes thoroughly I created a diamond graph of sbt plugins:

            sbt-plugin-example-diamond
                       / \
sbt-plugin-example-left   sbt-plugin-example-right
                       \ /
            sbt-plugin-example-bottom

I published the artifacts to Maven Central:

Depending on the version of sbt-plugin-example-diamond, from 0.1.0 to 0.5.0, different parts have migrated to the new pattern, and there can be conflict resolution on sbt-plugin-example-bottom. See the scripted dependency-management/sbt-plugin-diamond for more details.

sbt-plugin-example-diamond:0.5.0 is fully migrated to the valid Maven pattern. Hence Maven can resolve its full graph of dependencies, and resolve the conflict on sbt-plugin-example-bottom.

The Maven-style publication is tested in the scripted dependency-management/sbt-plugin-publish.

Next Steps

Once sbt 1.8 is EOL we can drop the publication of the old Maven pom and artifact.
If ever we drop the support of extra-attributes in Maven, we will still be able to resolve sbt plugins as org.example:example_2.12_1.0:1.0.0. We probably still want to resolve older plugins though, so it may not be possible to remove the resolution fallback and the extra-attributes mechanism.
In sbt 2.x we should drop all this complexity and use the CrossVersion mechanism to publish sbt plugin, similarly to Scala.js or Scala Native artifacts. The cross version would then be something like _sbt2_3.
After sbt 1.x in EOL we can drop the support of extra-attributes in Coursier

main/src/main/scala/sbt/Defaults.scala

sbt-app/src/sbt-test/dependency-management/sbt-plugin-diamond/test

sbt-app/src/sbt-test/dependency-management/sbt-plugin-publish/build.sbt

julienrf · 2022-12-21T16:13:29Z

sbt-app/src/sbt-test/dependency-management/sbt-plugin-publish/test

+
+> testProject / checkUpdate
+> set testProject / useCoursier := false
+> testProject / checkUpdate


Do we cover publishing to ivy repositories? (e.g. publishLocal)

Do we cover resolving a plugin that we just published?

That's a great question; I thought the Ivy pattern and approach is entirely different from that of SBT+Maven?

I expanded the scripted sbt-plugin-publish to test the publication to .ivy/local and resolution from it.

For the Ivy publication, there are two cases, depending on the value of publishMavenStyle.

publishMavenStyle := false: Strictly no changes, we publish:

org.example/example/scala_2.12/sbt_1.0/0.1.0-SNAPSHOT/ ├── docs │ └── example-javadoc.jar ├── ivys │ └── ivy.xml ├── jars │ └── example.jar └── src └── example-sources.jar

publishMavenStyle.value := true: We publish the legacy and new poms, and all the associated artifacts.

org.example/example/scala_2.12/sbt_1.0/0.1.0-SNAPSHOT/ ├── docs │ ├── example_2.12_1.0-javadoc.jar │ └── example-javadoc.jar ├── ivys │ └── ivy.xml ├── jars │ ├── example_2.12_1.0.jar │ └── example.jar ├── poms │ ├── example_2.12_1.0.pom │ └── example.pom └── src ├── example_2.12_1.0-sources.jar └── example-sources.jar

In the test, we do a publishLocal on a project that is configured for Maven, so it is a Maven-style publication to Ivy.

sbt plugins were published to an invalid path with regard to Maven's specifictation. As of sbt 1.9 we produce two POMs, a deprecated one and a new one, that is valid with regard to Maven resolution. When resolving, we first try to resolve the new valid POM and we fallback to the invalid one if needed. In the new POM format, we append the sbt cross-version to all artifactIds of sbt plugins. This is because we want Maven to be able to resolve the plugin and all its dependencies. When parsing it, we remove the cross-version suffix so that the result of parsing the valid POM format is exactly the same as parsing the deprecated POM format. Hence conflict resolution happens as intended. More details can be found at sbt/sbt#7096

julienrf

Thank you @adpi2 for all the PRs and the detailed explanations. I think this is a good approach.

ScalaWilliam · 2023-02-06T22:09:02Z

@adpi2 was just wondering what those failing tests are, do you have much idea? It seems they would get fixed by stopping the cache upload but then I am not sure if that's what we want.

sjrd · 2023-02-06T22:32:16Z

@adpi2 is off for a few weeks. There might be some delay here.

As of sbt 1.9, we publish deprecated and valid poms. In this test we check that sbt resolve the valid pom of an sbt plugin and fallback to the deprecated pom if the valid pom cannot be found.

For an sbt plugin, we publish two POM files, the legacy one, and the new Maven compatible one. The name of the new POM file contains the sbt cross-version _2.12_1.0. The format of the new POM file is also slightly different, because we append the sbt cross-version to all artifactIds of sbt plugins. Hence Maven can resolve the new sbt plugin POM and its dependencies. When resolving an sbt plugin, we first try to resolve the new Maven POM and if it fails we fallback on the legacy one. When parsing the new POM format, we remove the sbt cross-version from all artifact IDs so that there is no mismatch between old and new format of dependencies.

adpi2 · 2023-02-23T08:27:40Z

@eed3si9n I need a release of sbt/librarymanagement#409 to merge this PR, maybe just a milestone version. Let me know if I can help.

eed3si9n · 2023-02-26T23:07:52Z

https://github.com/sbt/librarymanagement/releases/tag/v1.9.0-M1 should be on its way to Maven Central.

adpi2 · 2023-02-27T08:52:59Z

@eed3si9n Something may have gone wrong, since I cannot find it: https://repo1.maven.org/maven2/org/scala-sbt/librarymanagement-core_2.12/

eed3si9n · 2023-02-27T16:46:46Z

@eed3si9n Something may have gone wrong, since I cannot find it: https://repo1.maven.org/maven2/org/scala-sbt/librarymanagement-core_2.12/

My bad. I forgot to release it from Sonatype staging. It should be there in 10 ~ 30 min.

adpi2 · 2023-02-28T09:07:27Z

@eed3si9n This PR is ready for a final review.

eed3si9n

Thanks!

ScalaWilliam · 2023-02-28T20:59:44Z

This is super exciting. When is the planned release date @eed3si9n ?

@sjrd - can't wait for the scala-js SBT plug-in once this is out :-)

sbt 1.9.0-M1 contains the new sbt plugin publishing mechanism. It dual-publishes sbt plugins with Ivy-style paths (invalid in Maven, although it works out when released from sbt) and valid Maven paths. See sbt/sbt#3410 and sbt/sbt#7096.

sbt 1.9.0-RC1 contains the new sbt plugin publishing mechanism. It dual-publishes sbt plugins with Ivy-style paths (invalid in Maven, although it works out when released from sbt) and valid Maven paths. See sbt/sbt#3410 and sbt/sbt#7096.

sbt 1.9.0 contains the new sbt plugin publishing mechanism. It dual-publishes sbt plugins with Ivy-style paths (invalid in Maven, although it works out when released from sbt) and valid Maven paths. See sbt/sbt#3410 and sbt/sbt#7096.

This was referenced Dec 21, 2022

Try resolve sbt plugin from valid Maven pattern coursier/coursier#2633

Merged

Try resolve sbt plugin from valid Maven pattern sbt/librarymanagement#409

Merged

The maven pattern for sbt plugins is invalid based on the spec #3410

Closed

adpi2 force-pushed the sbt-plugins-maven-path branch from e178312 to 550f3d1 Compare December 21, 2022 13:19

julienrf reviewed Dec 21, 2022

View reviewed changes

adpi2 force-pushed the sbt-plugins-maven-path branch 4 times, most recently from b2f1dbe to bec0d43 Compare December 27, 2022 09:45

julienrf approved these changes Dec 29, 2022

View reviewed changes

hamnis mentioned this pull request Jan 5, 2023

Support new publishing coordinates for plugins arktekk/sbt-aether-deploy#76

Closed

adpi2 force-pushed the sbt-plugins-maven-path branch 2 times, most recently from 024e5ac to 764534a Compare January 25, 2023 12:52

adpi2 force-pushed the sbt-plugins-maven-path branch from 764534a to 1f61f28 Compare February 22, 2023 08:40

adpi2 added 3 commits February 22, 2023 10:09

Add scripted to test the sbt plugin resolution

8e64caa

As of sbt 1.9, we publish deprecated and valid poms. In this test we check that sbt resolve the valid pom of an sbt plugin and fallback to the deprecated pom if the valid pom cannot be found.

Bump lm-coursier-shaded to 2.0.16

d05913f

adpi2 force-pushed the sbt-plugins-maven-path branch 2 times, most recently from 550933a to d05913f Compare February 23, 2023 08:16

Bump librarymanagement to 1.9.0-M1

ada716f

adpi2 marked this pull request as ready for review February 28, 2023 09:05

adpi2 requested a review from eed3si9n February 28, 2023 09:07

eed3si9n approved these changes Feb 28, 2023

View reviewed changes

eed3si9n merged commit 894789c into sbt:1.9.x Feb 28, 2023

sjrd mentioned this pull request Mar 8, 2023

Upgrade to sbt 1.9.0. scala-js/scala-js#4827

Merged

mdedetrich mentioned this pull request May 4, 2023

sbt 1.9.0-RC1 failing with checksum files must be unique #7233

Closed

adpi2 mentioned this pull request May 4, 2023

[1.9.x] Fix #7233: copy artifact files of sbt plugin #7234

Merged

mdedetrich mentioned this pull request May 8, 2023

Conflicting cross-version suffixes with SBT 1.9.0-RC2 #7243

Closed

sjrd mentioned this pull request Jun 3, 2023

Upgrade to sbt 1.9.0. portable-scala/sbt-platform-deps#13

Merged

adpi2 mentioned this pull request Sep 11, 2024

[2.x] Publish sbt plugins using platform prefix #7655

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[1.9.x] Smooth transition to valid Maven pattern of sbt plugins #7096

[1.9.x] Smooth transition to valid Maven pattern of sbt plugins #7096

Uh oh!

adpi2 commented Dec 21, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julienrf Dec 21, 2022

Uh oh!

ScalaWilliam Dec 23, 2022

Uh oh!

adpi2 Dec 27, 2022

Uh oh!

julienrf left a comment

Uh oh!

ScalaWilliam commented Feb 6, 2023

Uh oh!

sjrd commented Feb 6, 2023

Uh oh!

adpi2 commented Feb 23, 2023

Uh oh!

eed3si9n commented Feb 26, 2023

Uh oh!

adpi2 commented Feb 27, 2023

Uh oh!

eed3si9n commented Feb 27, 2023

Uh oh!

adpi2 commented Feb 28, 2023

Uh oh!

eed3si9n left a comment

Uh oh!

ScalaWilliam commented Feb 28, 2023

Uh oh!

Uh oh!

[1.9.x] Smooth transition to valid Maven pattern of sbt plugins #7096

[1.9.x] Smooth transition to valid Maven pattern of sbt plugins #7096

Uh oh!

Conversation

adpi2 commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Associated PRs

In depth

Generating the valid pom file

Resolving an sbt plugin

Tests

Next Steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julienrf Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

ScalaWilliam Dec 23, 2022

Choose a reason for hiding this comment

Uh oh!

adpi2 Dec 27, 2022

Choose a reason for hiding this comment

Uh oh!

julienrf left a comment

Choose a reason for hiding this comment

Uh oh!

ScalaWilliam commented Feb 6, 2023

Uh oh!

sjrd commented Feb 6, 2023

Uh oh!

adpi2 commented Feb 23, 2023

Uh oh!

eed3si9n commented Feb 26, 2023

Uh oh!

adpi2 commented Feb 27, 2023

Uh oh!

eed3si9n commented Feb 27, 2023

Uh oh!

adpi2 commented Feb 28, 2023

Uh oh!

eed3si9n left a comment

Choose a reason for hiding this comment

Uh oh!

ScalaWilliam commented Feb 28, 2023

Uh oh!

Uh oh!

adpi2 commented Dec 21, 2022 •

edited

Loading