kotlin improvements: debugger and compilation #349

dkimitsa · 2019-02-15T09:12:52Z

NOTE: this PR is intended for 2.3.7-SNAPSHOT as introduces changes in compilation pipeline.
currently created for wide testing

This PR changes following:

reworked method resolution of variables for Debugger

post in blog
Problem: Soot splits variables slot on assignment for type resolution and not always it able to pack them back. As result during method flow variable can be located in different locals (memory addresses). This affects both Java and Kotlin.
Before: it was supposed that variable will be located at single memory address. that causes cases when variable was not resolved for debugger (as was resolved to multiple locals/memory location);
Now: for each instruction visible variable scope is resolved. As result variable debugger can handle splitter variable case. This variable scope index is saved in debug information in Column index (altogether with line number);
Changes for this:

Soot was reworked. Previous variable resolution was dropped. Instead added RoboVM specific variable Packing code. Also added visible variable scope;
Hooks infrastructure changed to provide current instruction pointer (PC) in each frame. This allows to calculate frame offset from method start and visible variable slice at this offset for debugger.
LLVM support code updated to fetch column number to build offset to variable slice map;
Problems: some kotlin libraries has broken debug information as result some variables will not be resolved. post about this

added SMAP support for debugger

details in blog
Kotlin injects inline function in class files that messes line numbers information. But it also provides SMAP to allow get original line numbers. Support for SMAP was added

allowing soot to split slots dedicated for parameters

details in blog
Kolin re-uses parameter slots for local and this was not expected by RoboVM code. As result soot was not able to resolve local type constraints and crashed. Changes were done in Soot but also reflects debugger plugin.
This fixes #220

build instructions for MobiVM team

pull changes from my Soot repo, build and deploy 2.5.0-6
build llvm library and commit it (intentionally didn't commit it as binaries has to be build in single clean env due security reasons)

pre-build Idea plugin

rebuild IntelliJ Idea plugin (2.3.8-SNAPSHOT) is available for testing

…eference debug information)

…to have Dwarf* prefix. Just to make difference with all kinds of debug information

…find offset from method start)

- for each instruction variable slice is generated (list of visible variables at this instruction) - variables are being resolved not by line number of call stack but by PC - for this special table of pc to method start is generated which point to variables visible at specific PC

…ns in debug mode) for case when type of local doesn't match debug information. its kotlin case

* debug locations that were throwing exceptions removed * cleanup

… added. as debug version of llvm has assertion that checks for array type and will crash

…lving variables. As kotlin reuses arguments slots and this made mess. Now only relies on debug information provided in class file

…nymore

…first instruction of line, not before last. othervise complex expressions will be half solved when BP reaches.

dkimitsa · 2019-02-15T10:18:24Z

btw, RoboVM cache has to be wiped.

… different symbols: one for saving offset and second for emitting it. otherwise it crashes on assert due symbol was used reason in debug builds

florianf · 2019-03-17T08:06:06Z

@dkimitsa should we merge this now, since the release is done?

dkimitsa · 2019-03-17T09:35:57Z

@florianf yes, snapshot will give better feed back.
but is is not just a merge -- there more actions to be done, check first comment

Franjoo · 2019-03-17T20:25:36Z

Are there any plans when this feature will be finished? Thanks!

florianf · 2019-04-03T18:37:44Z

@Tom-Ski I merged the upstream changes in the soot repo, can you release a new version to sonatype?

florianf · 2019-04-19T18:34:54Z

Heads up: this is not forgotten, still waiting for @Tom-Ski to push a new soot release.

dkimitsa · 2019-04-23T19:07:37Z

2.3.7-SNAPSHOT Idea plugin with recent fixes is available for testing

dkimitsa · 2019-04-24T07:33:44Z

@florianf @Tom-Ski how about releasing 2.3.7 with fixes it has ? to have issue-less release version. and them moving forward with this stuff ?

florianf · 2019-05-11T10:44:30Z

@dkimitsa : Fine for me, currently only @Tom-Ski can do a release.

Dunemaster · 2019-05-14T13:08:34Z

compiler/compiler/pom.xml

+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-compiler-plugin</artifactId>
+        <version>2.3.2</version>


@dkimitsa, why so ancient maven-compiler-plugin?

there is lot of deprecated code, we hadn't touch it yet as long as it keep working

Franjoo · 2019-05-14T16:19:50Z

After I ran in some issues related to #220 and #349, I luckily got some help from @dkimitsa who provided the 2.3.7 Snapshot. So I want to give some testing feedback:

robovm used in a libgdx environment

the plugin solves:

1. error compiling kotlin coroutines (#220)

(https://gist.github.com/Franjoo/bfe600ae7e776a32d62919b98b7a7bef)

2. can not save and run roboVM on iOS simulator due to missing / unsafed run configuration

notes:

1. in libgdx projects it is needed to remove android module since there seems to be issues with facet configurations in workspace.xml

2. Got compilation error when using 'implementation' instead of 'compile' using robovm in build.gradle

3. Debugger was tested and works well

4. plugin was tested in large(r) libgdx project and works as expected (cocoutines, ios simulator run configuration, debugger).

dkimitsa · 2019-05-14T19:38:48Z

hi @Franjoo, thank you for providing feedback.

eusilin · 2019-07-08T12:28:44Z

It seems there are quite a few critical fixes in this PR.
Any ideas when it will be pulled in?
Can I get a snapshot version?

dkimitsa · 2019-07-11T09:29:19Z

have merged with master and re-build snapshot (link if first comment)

dkimitsa added 12 commits June 22, 2018 14:07

LLVM: line number info extended with column number (will be used to r…

94c4cab

…eference debug information)

LLVM, debugger: debug information retrieved from object file renamed …

c15c0be

…to have Dwarf* prefix. Just to make difference with all kinds of debug information

* hooks updated to include PC as part of call stack (will be used to …

9867482

…find offset from method start)

* removed exception and replaced with trace to log (in case RoboVM ru…

fe79318

…ns in debug mode) for case when type of local doesn't match debug information. its kotlin case

* reverted commited by mistake debug configuration for VM libraries

23fea99

* debugger: added support for Kotlin SMAP

45042f1

* debug locations that were throwing exceptions removed * cleanup

Merge branch 'master' into debugger_local_resolution

e97fab2

* always append to @llvm.used as array even if only element has to be…

fe7f90e

… added. as debug version of llvm has assertion that checks for array type and will crash

* variable slicer reworked to ignore parameter information while reso…

511aaac

…lving variables. As kotlin reuses arguments slots and this made mess. Now only relies on debug information provided in class file

* fixed to support recent builds, don't build 32bit version for mac a…

ecf7053

…nymore

* fixed instrumentation of instruction. hook has to be called before …

c86b054

…first instruction of line, not before last. othervise complex expressions will be half solved when BP reaches.

dkimitsa mentioned this pull request Feb 15, 2019

Exception in Lambda transformation using modified soot #220

Closed

* reworked emitting .spfpoffset for arm targets in the way to use two…

02248d1

… different symbols: one for saving offset and second for emitting it. otherwise it crashes on assert due symbol was used reason in debug builds

Merge branch 'master' into debugger_local_resolution

c6b5fa9

Dunemaster reviewed May 14, 2019

View reviewed changes

dkimitsa mentioned this pull request Jun 30, 2019

Yet another debugger crash #388

Closed

3 tasks

Merge branch 'master' into debugger_local_resolution

8e7dac5

Tom-Ski merged commit 8a1b02e into MobiVM:master Jul 11, 2019

dkimitsa deleted the debugger_local_resolution branch August 26, 2019 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

kotlin improvements: debugger and compilation #349

kotlin improvements: debugger and compilation #349

Uh oh!

dkimitsa commented Feb 15, 2019 •

edited

Loading

Uh oh!

dkimitsa commented Feb 15, 2019

Uh oh!

florianf commented Mar 17, 2019

Uh oh!

dkimitsa commented Mar 17, 2019

Uh oh!

Franjoo commented Mar 17, 2019 •

edited

Loading

Uh oh!

florianf commented Apr 3, 2019

Uh oh!

florianf commented Apr 19, 2019

Uh oh!

dkimitsa commented Apr 23, 2019

Uh oh!

dkimitsa commented Apr 24, 2019

Uh oh!

florianf commented May 11, 2019

Uh oh!

Dunemaster May 14, 2019

Uh oh!

dkimitsa May 14, 2019

Uh oh!

Franjoo commented May 14, 2019

Uh oh!

dkimitsa commented May 14, 2019

Uh oh!

eusilin commented Jul 8, 2019

Uh oh!

dkimitsa commented Jul 11, 2019

Uh oh!

Uh oh!

kotlin improvements: debugger and compilation #349

kotlin improvements: debugger and compilation #349

Uh oh!

Conversation

dkimitsa commented Feb 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

reworked method resolution of variables for Debugger

added SMAP support for debugger

allowing soot to split slots dedicated for parameters

build instructions for MobiVM team

pre-build Idea plugin

Uh oh!

dkimitsa commented Feb 15, 2019

Uh oh!

florianf commented Mar 17, 2019

Uh oh!

dkimitsa commented Mar 17, 2019

Uh oh!

Franjoo commented Mar 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

florianf commented Apr 3, 2019

Uh oh!

florianf commented Apr 19, 2019

Uh oh!

dkimitsa commented Apr 23, 2019

Uh oh!

dkimitsa commented Apr 24, 2019

Uh oh!

florianf commented May 11, 2019

Uh oh!

Dunemaster May 14, 2019

Choose a reason for hiding this comment

Uh oh!

dkimitsa May 14, 2019

Choose a reason for hiding this comment

Uh oh!

Franjoo commented May 14, 2019

1. error compiling kotlin coroutines (#220)

2. can not save and run roboVM on iOS simulator due to missing / unsafed run configuration

1. in libgdx projects it is needed to remove android module since there seems to be issues with facet configurations in workspace.xml

2. Got compilation error when using 'implementation' instead of 'compile' using robovm in build.gradle

3. Debugger was tested and works well

4. plugin was tested in large(r) libgdx project and works as expected (cocoutines, ios simulator run configuration, debugger).

Uh oh!

dkimitsa commented May 14, 2019

Uh oh!

eusilin commented Jul 8, 2019

Uh oh!

dkimitsa commented Jul 11, 2019

Uh oh!

Uh oh!

dkimitsa commented Feb 15, 2019 •

edited

Loading

Franjoo commented Mar 17, 2019 •

edited

Loading