Incremental Compilation #334

dthommes · 2018-10-16T15:42:46Z

As discussed in the Gitter channel, I have added incremental compilation capabilities to RoboVM's compiler for the case that no classes change between runs. This speeds up the build process for scenarios where you only modify resources or simply restart the app for hitting a certain break point in debug mode. Here are some details about the changes:

Changes in RoboVM Compiler

The compiler now writes a file named classpath_dependencies.txt into the build folder. This file contains information about the classes and jar files that were used to build the Main binary. On the next run (and if this file is still available), the compiler compares the creation dates of the files mentioned in this file and the Main binary. Only, if one of the given files is newer, recompilation is triggered.
In a second file (config.xml), the compiler's configuration is written into the build folder. With the next run, the configurations are compared. If there are changes (e.g. forcelink settings), recompilation is triggered.
The file assetcatalog_dependencies is now created by XCode's asset tool. With this file, it is possible to check whether assets have been changed between runs. It would be possible to skip asset packaging, if there are no changes. This is not implemented, yet.
The compiler will log a new message that is related to the incremental build decision.

These changes only take effect, if RoboVM's build folder is not cleaned between runs, which was the default case for all plugins before the below mentioned changes.

Changes in IDEA Plugin

To enable the new incremental build, I have removed the step in the IDEA plugin that cleans the robovm-build folder for each build. I have added a new menu entry in the RoboVM menu (Clean Build Folders) to do this manually instead. It was not possible to let the plugin run the clean process as part of IntelliJ's default clean action (Build -> Rebuild Project).
I have renamed the Menu Entry RoboVM -> Clean cache to RoboVM -> Clean Global Cache to make a clear distinction between the actions.
Timestamp logging was added in the RoboVM logs to get a better idea of the bottlenecks within the build process. This can be used to further improve build times in the future.

Tests

IDEA + Gradle: I have extensively tested the behavior of the incremental build with IDEA. There were no problems so far, cleaning between runs is typically not necessary.
IDEA + Maven: I have tested a Maven project with IDEA. It works correctly like the above setup (after you have fixed the source settings in pom.xml)
Eclipse: I have tested the Eclipse plugin. It still works like before because it keeps cleaning the RoboVM build folder between builds. It may be an option for the future to add IDEA-like behavior here. This would require a UI element for cleaning as mentioned above.
Gradle: The Gradle plugin works like before, as it also cleans the build folder with each run.
Maven: I was not able to build an iOS app with Maven on the command line, as there are missing dependencies to the RoboVM runtime. This does not seem to be related to this PR but is a general problem with Maven.

Speed-Up
For our extensive application with about 7000 classes, restarting the app without changes can now be done in 14 instead of 39 seconds.

…nges in input jar or class files

…iler active. Added timestamp for logging to spot bottlenecks.

# Conflicts: # compiler/compiler/src/main/java/org/robovm/compiler/AppCompiler.java

florianf · 2018-10-20T08:20:15Z

Thanks! Looks good to me, @dkimitsa what do you think?

dkimitsa · 2018-10-22T13:18:06Z

@florianf hi, will take a look tomorrow. thanks

florianf · 2018-11-01T19:37:23Z

Looks good to me, will merge in the next days if no one has objections.

…ersAction

dthommes · 2018-11-01T21:22:31Z

Hi Florian, that's great. I've been working with this version of the plugin in the last weeks and can still confirm that there are no problems.

I just saw, that the required copyrights were missing and one was mistakenly copied into a new class. Fixed that.

Best wishes!

obigu · 2018-11-01T21:39:47Z

@dthommes Haven't spent time looking in detail so I may have missed something obvious but could you please clarify how it improves the existing incremental compilation? RoboVM doesn't recompile every time but only when changes to classes are made, isn't that incremental compilation? Thanks.

dthommes · 2018-11-01T23:34:27Z

@obigu
You're right, there is incremental compilation available today. The compiler caches object code of external libraries. The compilation process looks like this (highly simplified):

java->class
class->soot
soot->llvm bitcode
llvm bitcode->object code
treeshaking
linking: object code->main binary

In short: This PR is about skipping steps 2-4 for classes in your project and steps 5-6 for all classes.

Longer info:
While RoboVM caches soot (.info) and object code (.o) from step 4 for external libraries (in the ~./robovm folder), it executed steps 2 to 4 for all classes in the actual project (as the build dir was deleted each time). Additionally, it executed steps 5-6 for all classes of an app without checking, whether this is required. For an app with several thousand classes, this takes many seconds.

Being incremental means, that the compiler should only execute the compilation process starting with a certain step, when the input for the same step has changed since the last run.

So let's say, this PR adds additional incremental behavior to the existing one.

obigu · 2018-11-02T01:05:40Z

Cool, thanks for the explanation, this is actually an improvement for the very specific (but quite common) scenarios of not changing any class.

One question, what happens when MobiVM version is updated? Will users need to remember to manually clean the build folder after the update?

dkimitsa · 2018-11-02T06:43:13Z

@dthommes

While RoboVM caches soot (.info) and object code (.o) from step 4 for external libraries (in the ~./robovm folder), it executed steps 2 to 4 for all classes in the actual project (as the build dir was deleted each time). Additionally, it executed steps 5-6 for all classes of an app without checking, whether this is required. For an app with several thousand classes, this takes many seconds.

there is a correction, all project classes.o go to ~/.robovm/cache folder altogether with class info as well. robovm-build is used for pre-deployment and linking activities.

probably your changes affects linking, deployment preparation and code sign. I will check code later today.

but first comment from my side:

this functionality should have on/off switch;
it is off by default

dthommes · 2018-11-05T09:45:57Z

@obigu
When updating MobiVM, you're getting a newer version of the robovm-rt.jar. The compiler will detect the change an trigger recompilation.

@dkimitsa
Thanks for the clarification. I wasn't aware that the object files for the project are in the global cache as well. However, this does not affect the functionality of this PR. It means, we're only skipping steps 5 and 6 from the above list, which is fairly effective.

Regarding switching the functionality of this PR: The compiler already reads the "clean flag" that can be given via the Config (org.robovm.compiler.config.Config#isClean()) and will recompile, if it is set. However, we don't have a GUI element for this in IntelliJ. Where would you see such an element? In a (new) preferences pane or as part of the run configuration?

dkimitsa · 2018-11-05T09:53:16Z

@dthommes isClean is just to removing build folder. but I mean to have this functionality completely off as it is quite experimental and will have side effects.
If someone wish to use it -- it might be enabled with switch. but default it should be off

dthommes · 2018-11-05T10:00:27Z

@dkimitsa Okay, so this might be a global setting and should be in a preferences pane, right?

dkimitsa · 2018-11-05T10:03:22Z

it would be enough to have it config

dthommes · 2018-11-05T10:08:21Z

Sorry, I don't understand. Is there a way to modify the config anyways?

dkimitsa · 2018-11-05T10:16:33Z

sorry for confusion, you mentioned that there is org.robovm.compiler.config.Config#isClean() that will force recompile.

but my point is that functionality added by this PR should have options to be turned on or off. and it shall be off by default. and it might be good to have it as option in config, similar to tree shaker option.

dthommes · 2018-11-05T10:22:26Z

Ah, okay. Then I would add an element for robovm.xml to optionally turn the feature on (being off by default).

… incremental compilation

dthommes · 2018-11-05T16:44:57Z

There you go: Just add

<smartCompile>true</smartCompile>

to your robovm.xml to enable the new feature. Otherwise, it will be disabled.

Get an impression of its effect on the restart time here:
https://www.youtube.com/watch?v=w3PQl2H2ySU

obigu · 2018-11-05T16:48:17Z

Thanks @dthommes . I think it's a nice addition and I like the fact that for the moment it's only opt-in via configuration (while it's experimental). My only minor criticism would be that as confirmed by @dkimitsa this change is not really about compilation (or incremental compilation) but about skipping linking, code sign, etc... The name of the property smartCompile is a bit confusing imo. I think smartSkipRebuild or something on those lines would be more appropriate.

dthommes · 2018-11-05T17:05:18Z

Yes, if everybody agrees with smartSkipRebuild, it's no problem to rename it.
Nevertheless, it would be great to finish this PR soon, because there are quite a few other issues in the IDEA Plugin that I could fix then.

florianf · 2018-11-08T13:13:10Z

@dthommes : Please re-name it, I think the name is o.k. Then I'll merge the PR. Thanks a bunch!

dkimitsa · 2018-11-08T13:22:24Z

@florianf please keep it open for this weekend, as I had no time yet to look through it. sorry for delay

dthommes · 2018-11-08T19:08:39Z

Renamed as suggested.

dkimitsa · 2018-11-12T10:53:29Z

compiler/compiler/src/main/java/org/robovm/compiler/util/ToolchainUtil.java

@@ -253,6 +254,10 @@ public static void actool(Config config, File partialInfoPlist, File inDir, File
            }
        }

+        //Prepares for incremental build - not applied at this time.


could you please add comment what it affect to and how it works

Added a longer comment here.

dkimitsa · 2018-11-12T10:55:15Z

plugins/idea/src/main/java/org/robovm/idea/RoboVmPlugin.java

@@ -223,12 +231,16 @@ public void run() {
        });
    }

+    private static String getNowString() {


I would rename it to something like getFormattedTimeStamp just to keep it same as everywhere in the world

dkimitsa · 2018-11-12T10:56:40Z

plugins/idea/src/main/java/org/robovm/idea/RoboVmPlugin.java

    public static void logInfo(Project project, String format, Object... args) {
-        log(project, ConsoleViewContentType.SYSTEM_OUTPUT, "[INFO] " + format, args);
+        log(project, ConsoleViewContentType.SYSTEM_OUTPUT, "[INFO] " + getNowString() + format, args);


do we really need these timestamps at every day basics ? As it useful during development but just mess an output when you don't care. Probably it might be useful to use it in case when RoboVM is running in development mode

Yes, we need them. It enables everybody to see, when there's a problem. Additionally it has the potential to motivate other people to contribute. E.g. somebody could be motivated to contribute logic to skip the slow interface compilation or asset creation processes (example from my machine):

[INFO] 10:22:33.071 /Applications/Xcode.app/Contents/Developer/usr/bin/ibtool ... [INFO] 10:22:35.726 /Applications/Xcode.app/Contents/Developer/usr/bin/actool ... [INFO] 10:22:38.892 /* com.apple.actool.compilation-results */

So, let's keep them.

dkimitsa · 2018-11-12T10:59:46Z

compiler/compiler/src/main/java/org/robovm/compiler/AppCompiler.java

+        }
+
+        //Has the configuration changed between runs (e.g. forcelink)?
+        boolean configsEqual;


why just don't check timestamps of robovm.xml, robovm.properties and Info.plist.xml files ?
it will do same but will remove lot of code that was added/changed

Creating a dependency to robovm.xml and the like would imply, that you're always configuring RoboVM via these files. This will fall back on you, if you once decide to make a configuration via Gradle or Maven (used today by javafxports and desirable anyways IMHO as of single-source-of-truth principle).
In fact, today you're configuring parts of the build process via the IDEA run configuration (e.g. the architecture - and please don't let us take this information from the build path).
So I think, comparing the configurations this way is required.

dkimitsa · 2018-11-12T11:00:21Z

compiler/compiler/src/main/java/org/robovm/compiler/AppCompiler.java

+
+        //Check for newer input files compared to Main binary
+        File classPathsFile = new File(config.getTmpDir(), CLASSPATHS_FILENAME);
+        File binaryFile = new File(config.getTmpDir(), "Main");


There is a bug here: application name should be checked from config and not hardcoded to Main.

dkimitsa · 2018-11-12T11:02:17Z

compiler/compiler/src/main/java/org/robovm/compiler/AppCompiler.java

+     */
+    private boolean needsRecompilation(Config config) throws IOException {
+        if (!config.isSmartSkipRebuild()) {
+            config.getLogger().info("Rebuilding because smartSkipRebuild is disabled. Enable it by adding " +


please remove it as we should not advertise any configuration when it not requested. especially experimental ones.

dkimitsa · 2018-11-12T11:04:34Z

hi @dthommes
Thanks for PR, I've tested and it was working on simple project.
I've found one bug and put bunch of comments just over the sources. Please take a look.
Thanks

dthommes · 2018-11-12T16:07:29Z

@dkimitsa Thanks for the review. I think this clarified the situation.
@florianf Would be great to see this merged. Please let me know, where you'd expect me to add some documentation on this feature as I have now removed all hints from the logs and disabled the feature per default. Would be a pitty to see nobody using it.

dthommes · 2018-11-12T16:09:36Z

BTW: I have tested the latest changes by building RoboVM in a whole and testing the IDEA plugin. It should be working fine.

dkimitsa · 2018-11-12T16:16:20Z

@dthommes btw, have you tested it with gradle plugin

dthommes · 2018-11-12T16:41:39Z

Yes (see first comment).

dthommes · 2018-11-12T16:43:26Z

Additionally: Tested it with Maven a few days ago after fixing the errors from the wrong template. Works as expected as Maven cleans the build folder.

dkimitsa · 2018-11-17T13:33:42Z

plugins/templates/ios-single-view-no-ib/src/main/resources/archetype-resources/robovm.xml

@@ -12,4 +12,6 @@
            <directory>resources</directory>
        </resource>
    </resources>
+    <!-- Set to 'true' to trigger compilation and linking depending on changes in IDEA -->


another moment/bug-like:
please comment out <smartSkipRebuild>false</smartSkipRebuild> as when previous compiler is used (without IC) it crashes with exception
ElementException: Element 'smartSkipRebuild' does not have a match in class org.robovm.compiler.config.Config

I understand that new compiler will handle it but it is better to keep compatibility with older ones.
Probably better case would be to have it commented out with true state and comment saying to uncomment line bellow to enable

Fixed this.

…kipRebuild element

florianf · 2018-11-27T21:23:34Z

@dkimitsa Is this mergeable now?

dkimitsa · 2018-11-27T21:40:21Z

Yep, it works for me. Thanks

florianf · 2018-11-28T10:45:25Z

Merged, thanks guys!

dthommes · 2018-11-28T10:58:35Z

@dkimitsa @florianf Thank you!

dthommes added 6 commits September 28, 2018 18:16

Compiler: Introduced skipping rebuild for cases when there are no cha…

18f2717

…nges in input jar or class files

IDEA: Removed force clean to make new incremental feature of the comp…

0e9f0a5

…iler active. Added timestamp for logging to spot bottlenecks.

Compiler: Fixing config problem in incremental build

3634568

IDEA: Added new "Clean Build Folders" entry in RoboVM menu.

6bd90d6

Merge branch 'master' into fast-compiler-2

ee4330c

# Conflicts: # compiler/compiler/src/main/java/org/robovm/compiler/AppCompiler.java

Fixed an import in AppCompiler

21ad9ed

Added copyright information and corrected copyright in CleanBuildFold…

91a3b95

…ersAction

Introduced <smartCompile> element for robovm.xml to optionally enable…

e8488e1

… incremental compilation

Renamed <smartCompile> to <smartSkipRebuild>

9b151d5

dkimitsa reviewed Nov 12, 2018

View reviewed changes

dthommes added 4 commits November 12, 2018 15:41

Clarification for actool parameter that writes dependency information

418e25f

Renamed "getNowString" to "getFormattedTimeStamp"

14b3f50

Took executable's name from config and removed hard coded name.

f688370

Removed hint for the new functionality in log

46d3c3c

dkimitsa reviewed Nov 17, 2018

View reviewed changes

Downgrading issue: Previous version failed when trying to load smartS…

d094283

…kipRebuild element

florianf merged commit 52c5d6b into MobiVM:master Nov 28, 2018

Incremental Compilation #334

Incremental Compilation #334

Uh oh!

Conversation

dthommes commented Oct 16, 2018

Uh oh!

florianf commented Oct 20, 2018

Uh oh!

dkimitsa commented Oct 22, 2018

Uh oh!

florianf commented Nov 1, 2018

Uh oh!

dthommes commented Nov 1, 2018

Uh oh!

obigu commented Nov 1, 2018

Uh oh!

dthommes commented Nov 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

obigu commented Nov 2, 2018

Uh oh!

dkimitsa commented Nov 2, 2018

Uh oh!

dthommes commented Nov 5, 2018

Uh oh!

dkimitsa commented Nov 5, 2018

Uh oh!

dthommes commented Nov 5, 2018

Uh oh!

dkimitsa commented Nov 5, 2018

Uh oh!

dthommes commented Nov 5, 2018

Uh oh!

dkimitsa commented Nov 5, 2018

Uh oh!

dthommes commented Nov 5, 2018

Uh oh!

dthommes commented Nov 5, 2018

Uh oh!

obigu commented Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dthommes commented Nov 5, 2018

Uh oh!

florianf commented Nov 8, 2018

Uh oh!

dkimitsa commented Nov 8, 2018

Uh oh!

dthommes commented Nov 8, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkimitsa commented Nov 12, 2018

Uh oh!

dthommes commented Nov 12, 2018

Uh oh!

dthommes commented Nov 12, 2018

Uh oh!

dthommes commented Nov 1, 2018 •

edited

Loading

obigu commented Nov 5, 2018 •

edited

Loading