Skip to content

Conversation

HGuillemet
Copy link
Collaborator

When running JPMS applications, JavaCPP users should only declare, in their module descriptor, the platform-independent module they need. For instance:

requires org.bytedeco.opencv

The platform-dependent native modules need to be added to the module graph with --add-modules command line argument when running the application, and when building an image with jlink.

Since the module path is automatically populated with the needed native jars by Maven (using the javacpp.platform property)
or by the Gradle plugin (using the javacppPlatform extra property), --add-modules ALL-MODULE-PATH is enough for running the application.

For building a JLink image with Maven, we can use the Apache Maven JLink plugin that automatically generates a --add-modules from the modules found in the module path.
But with Gradle, we have nothing similar. If we use the JavaFX Gradle plugin or the Badass jlink plugin, there is no such auto-generated --add-modules and thus the native modules are not added to the image.
If we configure Badass jlink plugin with --add-module ALL-MODULE-PATH, the image will contain all JavaCPP native modules
but also all the standard modules from the JDK, ruining most of the interest of building a custom runtime with jlink.

This PR adds the computation of a native module list that can be used in the Badass JLink plugin configuration to generate a proper --add-module argument.
It defines a javacpp extension to the platform plugin that contains the plaforms property and the nativeModules property.
The nativeModules property is computed during execution after the classes task (since it uses the runtime classpath, which can be modified during task execution), so must be added to the jlink task configuration in a doFirst block.
The platforms property replaces the javacppPlatforms extension property, but configuring the extra property is still supported for backward compatibility. Directly using the platforms property in a javacpp DSL block looks just a bit better.
More configurations for the JavaCPP plugin could be added later in this extension.
Here is an example of build script with kotlin syntax:

plugins {
    application
    id("org.bytedeco.gradle-javacpp-platform") version "1.5.8-SNAPSHOT"
    id("org.beryx.jlink") version "2.25.0"
}

javacpp {
  platforms = listOf("linux-x86_64")
}

dependencies {
    implementation("org.bytedeco:opencv-platform:4.5.5-1.5.7")
}

group = "org.bytedeco"
version = "1.0-SNAPSHOT"
description = "gradle-jlink-sample"

application {
    mainModule.set("org.bytedeco.sample")
    mainClass.set("org.bytedeco.sample.Application")
    applicationDefaultJvmArgs = listOf("--add-modules", "ALL-MODULE-PATH")
}

tasks.jlink {
  doFirst {
    jlink {
      options.set(listOf("--add-modules", javacpp.nativeModules))
      launcher {
        jvmArgs = listOf("--add-modules", javacpp.nativeModules)
      }
    }
  }
}

Note that the JavaFX gradle plugin doesn't allow to configure --add-modules for jlink, and is not needed anyway to run or link applications using JavaFX.

Add NativeModules
@saudet
Copy link
Member

saudet commented Mar 16, 2022

This doesn't sound like something that should be part of a plugin for JavaCPP. Wouldn't that be more appropriate in a more generic plugin for jlink like https://github.com/moditect/moditect-gradle-plugin ?

The platforms property replaces the javacppPlatforms extension property, but configuring the extra property is still supported for backward compatibility. Directly using the platforms property in a javacpp DSL block looks just a bit better.

The problem with properties from custom classes like that is that I haven't found a way to let users set them from the command line. Am I missing something?

@saudet
Copy link
Member

saudet commented Mar 16, 2022

From what I understand of how Gradle works, when inferModulePath isn't disabled, it should already be putting everything the module path, by default: https://docs.gradle.org/current/userguide/java_library_plugin.html

Why isn't this happening?

@saudet
Copy link
Member

saudet commented Mar 16, 2022

Ah, it looks like the badass plugin does its own inferencing instead of just using the module path from Gradle, see beryx/badass-jlink-plugin#198. I don't see the point of finding only the native modules related to JavaCPP when users probably just want to add all dependencies that they list anyway. Could you elaborate on the use case as to why this needs to be specific to JavaCPP modules instead of just taking everything from the module path?

@HGuillemet
Copy link
Collaborator Author

HGuillemet commented Mar 16, 2022

The problem with properties from custom classes like that is that I haven't found a way to let users set them from the command line. Am I missing something?

I haven't found either. But the platforms extension property is initialized with extra property javacppPlaform, if it exists, so -PjavacppPlaform=... still works.

From what I understand of how Gradle works, when inferModulePath isn't disabled, it should already be putting everything the module path, by default: https://docs.gradle.org/current/userguide/java_library_plugin.html

Why isn't this happening?

The module path does contain everything, it even contain too much (see below). The problem adressed by this PR is to
popupate the readable module graph, not the module path.

Could you elaborate on the use case as to why this needs to be specific to JavaCPP modules instead of just taking everything from the module path?

"Taking everything from the module path" is covered by the --add-modules special value ALL-MODULE-PATH.
This works, if the module path doesn't not contain more that what is sufficient.
The problem with the Badass plugin is that it systematically adds $JAVA_HOME/jmods to the module path, thus the whole jdk ends up in the image if we use --add-modules ALL-MODULE-PATH.
This might be considered as a bug. Since Java 10 we have this feature and the Badass and other plugins should probably rely on this feature and not add $JAVA_HOME/jmods. I'll raise an issue and see what the author thinks about it.
Another possible problem is with automatic modules, like the one from JavaFX, that must be in the dependencies, but not in the jlink module path. However it looks like the Badass plugin filters them out. If we use the Gradle module path computed by inferModulePath, it would contain the automatic modules.

I agree that this PR doesn't bring a very satisfactory solution for people needing to link with Gradle. For instance, the way it selects the modules to add : from the runtime paths and looking for "system.arch" in their name, is not rigorous.
The ideal solution for me would be to pre-extract the native libraries in the image runtime instead of adding the native jars.

@saudet
Copy link
Member

saudet commented Mar 17, 2022

I agree that this PR doesn't bring a very satisfactory solution for people needing to link with Gradle. For instance, the way it selects the modules to add : from the runtime paths and looking for "system.arch" in their name, is not rigorous. The ideal solution for me would be to pre-extract the native libraries in the image runtime instead of adding the native jars.

Right, so, like I tried to explain before, Android is also too dumb to support native libraries bundled in JAR files, but we can easily compensate for that by extracting the libraries manually using Gradle, and spoon-feeding the extracted directory to its plugin. I've added documentation for that at the bottom of the README.md file, it's literally just 10 lines of code! But you know 10 lines of code is way too much code for the Android team to add to their plugin. What they want us to do is publish AAR files instead of JAR files. And now OpenJDK wants us to publish native modules in JMOD files instead of JAR files, so it's like what the hell is this crap? It triples the size of binaries uploaded to Maven Central, and it triples the time we have to maintain them. I'm not going to play that game! Why not just support JAR files, you know? They work just fine, OSGi has been supporting native libraries in JAR files for like forever, and it works just perfectly fine thank you! There's absolutely no reason to deal with this idiocy. Since neither Google nor Oracle is interested in making Java a sane platform when it comes to native binaries, we need to figure out something on our own. For starters, I'm sure we can come up with a hack very similar to the one for Android, but for jlink, so please do give it a try. If you are absolutely sure there is no way to deal with this in a sane way, then let's continue complaining at OpenJDK. It's their problem, also because Android depends OpenJDK. Maybe they will listen, eventually, but the chances of this happening are greater if we are able to show them that JAR files are sufficient and that we don't need JMOD files.

/cc @jjh-aicas @johanvos @mikehearn @timothyjward

@HGuillemet
Copy link
Collaborator Author

In the meantime, what about generalizing your Android workaround and add some lines of code to this Gradle plugin that add the option of pre-extracting the native jars into the jlink images ?

@saudet
Copy link
Member

saudet commented Mar 18, 2022

If you know a way to make this work, let's have a look at that, yes. That's what I said could help us convince people at OpenJDK to start doing the right thing.

@HGuillemet HGuillemet changed the title Improve JPMS support Improve JPMS support: linking native jars into the jlink image Mar 20, 2022
@HGuillemet
Copy link
Collaborator Author

Here it is: #21

@HGuillemet HGuillemet closed this Mar 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants