Skip to content

Add better support for dependency management (aka BOM) #3097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 7, 2024

Conversation

alexarchambault
Copy link
Member

@alexarchambault alexarchambault commented Oct 16, 2024

What these changes are about

This PR changes things related to the so called Maven "BOMs" or "dependency management" support. It changes the way versions are picked for many dependencies of various JVM ecosystems (Android, Spark, Spring, …), hence the "sensitiveness" of these changes.

What BOMs are

BOMs are basically POM files listing versions to be used for dependencies. They look like this: this POM contains a dependencyManagement section, listing many dependencies. These dependencies' version and sometimes exclusions matter to us here.

A standard Maven dependency can pull a BOM either by declaring it as a "parent", like at the top of this POM, or by declaring an "import" dependency in its dependencies or dependencyManagement sections, like in this POM. Note that parents POMs can themselves have parents and "import" dependencies, whose values are taken into account too. Note also that POMs can also have a <dependencyManagement> section, acting as a BOM too, like this POM - BOM values don't always come from other POM files.

When added to a Maven project, Maven BOMs are basically expected to be dependency override lists. If project A has B:1.2 in its BOM, it is expected either: not to pull B (if neither A or its transitive dependencies depend on B), or to pull B version 1.2 (and no other version). No matter what version of B a transitive dependency of A depends on, if pulled via A, B should have version 1.2, because of A's BOM.

Also, it's common practice in Maven projects having a BOM not to put versions in dependencies in their dependencies section. These versions are supposed to be filled in with values from the BOM. We can see that clearly in this POM.

BOM support in coursier

Up to now, coursier used BOM values for the latter case (filling up empty versions of direct dependencies of the dependency that pulls the BOM), not the former (dependency overrides, applying to transitive dependencies too). Early checks of mine (around… 2015 😅) didn't display the former behavior, so I assumed all those BOM values where only meant for the latter case, when one pulls Maven dependencies.

This PR changes that: it uses the BOM values as "dependency overrides" too, applying to transitive dependencies.

Impact of those changes

One can see in the diff of this PR how many resolutions are impacted (or not impacted…). Many resolutions' results are hard-coded under modules/tests/shared/src/test/resources/resolutions. We can see in the diff here that Spark or Quarkus dependencies are impacted, among others. Overall, these changes are consistent with the new BOM values handling (all individual values I looked at in detail were fine at least).

Should these changes be enabled by default right now?

The new way of computing versions gives more consistent dependencies to me, more in line with what Maven users use when building their projects. These changes ought to be welcomed, yet they might surprise some users.

I'm thinking these could be enabled by default, but advertized loudly in release notes, and users need to be able to disable them if they want to.

Dependency overrides handling in build tools

BOMs give a "first class citizen" status to "dependency overrides". BOMs are dependency overrides that can be published, and pulled by downstream users.

In sbt for example, the values of dependencyOverrides are equivalent to BOM values. As a consequence, these can be published. These can go in a <dependencyManagement> section of a module POM file, like in this POM. Then, when users pull that sbt-published module, coursier applies the dependency overrides from the <dependencyManagement> section to transitive dependencies pulled by the module.

AFAIK, Mill doesn't have an equivalent target.

A simple setting like dependencyOverrides in sbt or an equivalent target in Mill only correspond to dependencyManagement sections in library POMs. They don't allow to publish shared dependency overrides via "parent POMs" and "dependency imports", nor to pull such overrides from Maven Central. Gradle apparently allows such uses.

@alexarchambault alexarchambault force-pushed the bom-experimentation branch 5 times, most recently from 64fffe6 to a1f37b3 Compare October 16, 2024 15:47
@alexarchambault alexarchambault force-pushed the bom-experimentation branch 6 times, most recently from 6cdf73c to 4d52c37 Compare October 23, 2024 11:51
@alexarchambault alexarchambault force-pushed the bom-experimentation branch 2 times, most recently from cc22c69 to a8173b4 Compare October 23, 2024 16:17
@alexarchambault alexarchambault changed the title BOM stuff experimentation Add better support for dependency management (aka BOM) Oct 23, 2024
@alexarchambault

This comment was marked as resolved.

@alexarchambault alexarchambault force-pushed the bom-experimentation branch 4 times, most recently from 5eec3c5 to 945798d Compare October 24, 2024 15:52
@alexarchambault alexarchambault force-pushed the bom-experimentation branch 3 times, most recently from 3bfd3d5 to c89f5b1 Compare October 24, 2024 21:11
@alexarchambault

This comment was marked as outdated.

@alexarchambault
Copy link
Member Author

cc @eed3si9n who dived a quite a bit in dependency resolver intricacies and @lefou who manifested interest for BOM support in #1390. Also cc-ing @lihaoyi and @adpi2 given this should impact both Mill and sbt, and @jin for rules_jvm_external

@lihaoyi
Copy link
Contributor

lihaoyi commented Oct 29, 2024

I haven't reviewed the code, but the PR description looks like exactly what we'd want, since it makes Coursier resolutions behave the same as Maven or Gradle or other tools

@alexarchambault
Copy link
Member Author

AFAIK, Mill doesn't have an equivalent target.

I'm probably going to have a look at this next: adding an equivalent to dependencyOverrides in Mill, and make sure its values are added to published POM files (and checking that these overrides are taken into account when coursier pulls these)

@lefou
Copy link
Collaborator

lefou commented Oct 29, 2024

In sbt for example, the values of dependencyOverrides are equivalent to BOM values. As a consequence, these can be published.

I think this is mostly true, but there is a small details of Maven dependency quirk, which we might have to be aware: I think Maven prioritizes direct dependency over dependencyManagement.

I think that matches my perception. Local dependency declarations override management overrides parent poms. There are a lot of precedence rules in Maven with regard to dependency management. I think the following page list some or all (?) of them:

https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#dependency-management

It also mentions a new packaging type bom (since Maven 4.0 and the POM Model 4.1.0).

In case Coursier will follow the Maven semantics, we should name the new task in Mill rather dependencyManagement than dependencyOverrides, as "overrides" implies a simple rule. Too simple.

@jin
Copy link

jin commented Oct 30, 2024

cc @shs96c for rules_jvm_external's BOM support.

We currently ask folks who need BOMs to switch to a different Aether based resolver (https://github.com/bazel-contrib/rules_jvm_external?tab=readme-ov-file#experimental-support-for-maven-bom-files), so it would definitely be nice to have this.

We will be happy to try out a build of coursier with our tests (https://github.com/bazel-contrib/rules_jvm_external/blob/3ea96498725cb21e3ac18490cc3f3b75db2259ca/MODULE.bazel#L618, https://github.com/bazel-contrib/rules_jvm_external/blob/master/MODULE.bazel#L432)

@alexarchambault
Copy link
Member Author

What needs to be done in Mill:

  • Decide, how we do declare managed dependencies? We should add a new task to CoursierModule. We need to agree on a good name. The type should be T[Seq[Dep]].
  • Decide, how we do import a BOM dependencies in import-scope? In Mill dependencies don't have scopes, but we manage multiple scope-like dependency sets: ivyDeps ~= compile-scope, runIvyDeps ~= runtime-scope, compileIvyDeps ~= provided-scope. (To import BOMs via ivyDeps, we need to find out, if Maven makes a distinction between these scopes and if there are any situations, where mixing compile and import scope for BOM management is problematic.) We better add a new task to import those BOMs. The task name might match the other new task name but add a suffix like Includes or Imports. The type is also T[Seq[Dep]].
  • How can we produce BOM artifacts? I think this should already work by changing the PublishModule.pomPackagingType to Pom. We just need to generate the <dependencyManagement> node in the pom task. Don't know what the equivalent to ivy is, if there is any.

@lefou Thanks for all these details. Let's discuss these Mill-specific points in an upcoming Mill issue or early PR that I'll open about this

@alexarchambault
Copy link
Member Author

I think this is mostly true, but there is a small details of Maven dependency quirk, which we might have to be aware: I think Maven prioritizes direct dependency over dependencyManagement.

@eed3si9n Thanks for mentioning this. I'm well aware of this point, coursier does follow this rule when using dependency management. I didn't mention it to keep my explanations simple, but this will have to be taken into account when generating POM files: in sbt, if dependencyOverrides versions take over libraryDependencies versions, the generated POM should only have the former versions, not the latter, in order for dependency resolution to use the same version

@alexarchambault alexarchambault marked this pull request as ready for review November 7, 2024 15:22
@alexarchambault
Copy link
Member Author

Going to merge this, with the feature enabled by default. I should mention it in the next release notes, but those changes can be easily disabled, and you might want to allow sbt / Mill / rules_jvm_external users to do so.

From the API, it can be disabled via the ResolutionParams, like

Resolve()
  .mapResolutionParams(_.withEnableDependencyOverrides(Some(false)))
// or
Fetch()
  .mapResolutionParams(_.withEnableDependencyOverrides(Some(false)))

On the CLI, --enable-dependency-overrides=false allows to disable it.

@alexarchambault
Copy link
Member Author

For scala-{library,reflect,compiler}, most of the time, these versions are just forced during dependency resolution. We might want to manage those via a dependency management section or even a BOM now 🤔 That BOM could even be managed by the Scala compiler team itself, especially for Scala 3, where the scala-library version (2.13 scala-library that Scala 3 pulls) cannot be mapped directly from the Scala 3 version.

@alexarchambault
Copy link
Member Author

alexarchambault commented Nov 7, 2024

cc @Gedochao, as Scala CLI resolves dependencies and publishes them too. IIRC, Scala CLI uses coursier via coursier-interface. Options to disable the feature here should be added to it too, to allow Scala CLI to disable it too. Maybe some BOM support could be added to Scala CLI too (see discussion above)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants