dist/tools: Add compile_like_murdock #18803

MrKevinWeiss · 2022-10-26T12:34:33Z

Contribution description

Helper script to pre-test murdock build conditions.

This is intended to compile like murdock giving control for only a subset of boards or applications. One can use this if only a limited change should be build tested. Defaults boards and apps are selected to be an early warning if something is wrong.

This should be used before triggering much larger murdock builds.

The following use cases are:

I made a change to something in the stm32 clocks... ./compile_like_murdock.py -c stm32

I changed a driver the DHT driver
./compile_like_murdock.py -a tests/driver_dht tests/saul

I changed a nucleo-f103rb board...
./compile_like_murdock.py -a tests/driver_dht tests/saul

Testing procedure

View the help...

./dist/tools/compile_test/compile_like_murdock.py -h

Do a dry run of the cpu...

./dist/tools/compile_test/compile_like_murdock.py -c stm -d

Play with each of the args, try adding a module to hello-world to get a module mismatch.

Issues/PRs references

MrKevinWeiss · 2022-10-26T12:35:03Z

ping @Teufelchen1 @benpicco

MrKevinWeiss · 2022-10-26T12:35:43Z

I would see if I can add a trick to also add apps with x module.

Teufelchen1 · 2022-10-26T13:40:04Z

dist/tools/compile_test/compile_like_murdock.py

+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-b", "--boards", nargs="+",


Choosing an invalid / not existing board does not result in an error. Is this intentional?

If you board is not available it just will skip it and should be somewhat apparent to the user, especially if it is a dry run.

We cannot easily know the boards that are available as this should also work with external boards.

I could add some logic if any board has not be selected in the whole run then maybe make a note at the end. Or I can add some less whimsical statistics such as tests run on each board (or pass/fail per board). This is kind of going against the idea of a simple wrapper around what was a few bash commands.

Teufelchen1 · 2022-10-26T13:46:08Z

dist/tools/compile_test/compile_like_murdock.py

+    parser.add_argument("-n", "--no-nightly", action="store_false",
+                        help=("Prevent running as nightly, by default NIGHTLY "
+                              "is set to 1, making more extensive testing."))
+    parser.add_argument("-a", "--apps", nargs="+",


Choosing an non-existent app results in a traceback. Since this will be most of the time a typo, I would suggest catching that exception and just print to the user something like: "Couldn't find the app FOO".
Keep in mind that the current code crashes after already running some apps when multiple had been given (depending on the order of apps). e.g. ./dist/tools/compile_test/compile_like_murdock.py -b slstk3400a -a examples/hello-world examples/typo-in-app fails after already running hello-world. Given that I would opt to check for existent either before running anything or not crashing but log afterwards which apps haven't been run and why. I prefer the first option.

dist/tools/compile_test/compile_like_murdock.py

MrKevinWeiss · 2022-10-27T09:20:15Z

dist/tools/compile_test/compile_like_murdock.py

+            print(f"{app: <30} {board: <30} FAIL: Kconfig module or pkg "
+                  "mismatch")
+        else:
+            print(app, board, "FAIL")


Missing the <30 spacing...

MrKevinWeiss · 2022-10-27T10:11:06Z

dist/tools/compile_test/compile_like_murdock.py

+        if "mismatch" in err.output:
+            print(f"{app: <30} {board: <30} FAIL: Kconfig hash mismatch")
+        elif lines[-3].startswith('< ') or lines[-3].startswith('> '):


Maybe these should be switched as module mismatches almost always means hash mismatches but not the other way around.

MrKevinWeiss · 2022-11-01T08:39:55Z

Anyone interested in this tool?

maribu · 2022-11-01T09:00:26Z

I do think this is convenient and a good addition to the workflow. We should however IMO be careful to not have to many competing solution to basically scratch roughly the same itch. Could there be some de-duplication performed between dist/tools/compile_test/compile_test.py, dist/tools/compile_and_test_for_board and this?

MrKevinWeiss · 2022-11-01T10:06:12Z

dist/tools/compile_and_test_for_board is for a single board, for running tests, and generate many files with results, and doesn't use murdocks compile command... I was thinking about that but it would likely result in much more than 100 LonwC (Lines of non-whimsical Code) and much more documentation and reading.

This was originally just a wrapper for a few lines of bash

export TEST_BOARDS=$(FEATURES_REQUIRED=cpu_stm32 make -C tests/pkg_tinyusb_cdc_msc info-boards-supported --no-print-directory)
for board in $TEST_BOARDS; do echo $board && /bin/bash -c "source .murdock; JOBS=8 compile tests/pkg_tinyusb_cdc_msc ${board}:gnu" | grep mismatch; done;

Probably the place it belongs is the dist/tools/compile_test/compile_test.py but again, would be quite some effort to merge the features of both. If you think that is worth it than I will rework it in exchange for a full review 😉

maribu · 2022-11-01T10:21:52Z

How about adding a documentation page instead that lists which tool does what?

maribu · 2022-11-01T10:59:13Z

How about adding a documentation page instead that lists which tool does what?

I mean instead of reworking. Apparently the use cases do differ enough to justify different implementations, but that was at least to me not obvious.

Sorry for being so sparse with words, I was in a rush 😅

MrKevinWeiss · 2022-11-01T11:34:48Z

No problem, I get not having time. Good point on the docs, I thought I could get away with just a docstring but will make it more official and add info on the other tools.

MrKevinWeiss · 2022-11-02T10:20:14Z

I actually think, for now, the docstring should suffice? This is only really mean to be used by maintainers/non-trivial contributors.

The compile_test.py is only referenced in 2 places.

In the .drone.yaml which we should probably remove as I don't think anyone uses it boards/*: remove unused FEATURES_MCU_GROUP variable #11670 (comment) and why keep something that we are not testing.
In the https://github.com/RIOT-OS/Release-Specs/blob/master/01-ci/README.md#task-01---compile-test which, again nobody would actually use and just open an "empty" PR to run the tests as running on a single computer would take some time.

If we really want to be careful with this tool I can just rename compile_test.py to _legacy_compile_test.py and compile_like_murdock.py to compile_test.py then invoke the legacy script if no arguments are added...

Maybe a different PR... Any opinions?

benpicco

This sounds pretty useful and it does not affect any existing scripts - so let's get this in!

Just one nit to make static tests happy:

dist/tools/compile_test/compile_like_murdock.py

jia200x · 2022-12-08T09:42:35Z

dist/tools/compile_test/compile_like_murdock.py

+import datetime
+
+
+_RATIOS = [


jia200x · 2022-12-08T13:48:48Z

dist/tools/compile_test/compile_like_murdock.py

+_RATIOS = [
+    ('digits of pi calculated', 4423213),
+    ('bitcoins mined', 0.000000077),
+    ('meters driven in a Telsa', 5),


Suggested change

('meters driven in a Telsa', 5),

('meters driven in a Tesla', 5),

oops very important!

Helper script to pre-test murdock build conditions. This is intended to compile like murdock giving control for only a subset of boards or applications. One can use this if only a limited change should be build tested. Defaults boards and apps are selected to be an early warning if something is wrong. This should be used before triggering much larger murdock builds. The following use cases are: I made a change to something in the stm32 clocks... ./compile_like_murdock.py -c stm32 I changed a driver the DHT driver ./compile_like_murdock.py -a tests/driver_dht tests/saul I changed a nucleo-f103rb board... ./compile_like_murdock.py -a tests/driver_dht tests/saul

riot-ci · 2022-12-08T17:24:43Z

Murdock results

✔️ PASSED

53d266e dist/tools: Add compile_like_murdock

Artifacts

Documentation preview

benpicco · 2022-12-08T18:40:14Z

bors merge

bors · 2022-12-08T19:01:45Z

Build succeeded:

Murdock
static-tests
tools-build

MrKevinWeiss · 2022-12-12T10:21:09Z

It seems that the -n arg doesn't actually turn nightlies off or something. I will look into it.

MrKevinWeiss · 2022-12-12T11:17:09Z

Actually it seems fine, I will have to ask @jia200x to test again on his setup...

github-actions bot added the Area: tools Area: Supplementary tools label Oct 26, 2022

benpicco requested review from kaspar030 and maribu October 26, 2022 12:40

Teufelchen1 reviewed Oct 26, 2022

View reviewed changes

MrKevinWeiss mentioned this pull request Oct 27, 2022

sys/ztimer: model ztimer_periph_lptimer for Kconfig #18783

Merged

MrKevinWeiss commented Oct 27, 2022

View reviewed changes

MrKevinWeiss mentioned this pull request Oct 27, 2022

Fix kconfig mismatch breaking nightlies #18805

Merged

MrKevinWeiss mentioned this pull request Dec 7, 2022

drivers/mrf24j40: add support for IEEE 802.15.4 Radio HAL #18472

Merged

benpicco reviewed Dec 7, 2022

View reviewed changes

dist/tools/compile_test/compile_like_murdock.py Outdated Show resolved Hide resolved

benpicco approved these changes Dec 7, 2022

View reviewed changes

jia200x reviewed Dec 8, 2022

View reviewed changes

dist/tools/compile_test/compile_like_murdock.py

import datetime

_RATIOS = [

Copy link

Member

jia200x Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤣

MrKevinWeiss force-pushed the pr/dist/compilelikemurdock branch from 585f259 to 5208c08 Compare December 8, 2022 12:14

jia200x reviewed Dec 8, 2022

View reviewed changes

MrKevinWeiss force-pushed the pr/dist/compilelikemurdock branch from 5208c08 to 53d266e Compare December 8, 2022 15:53

benpicco added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: skip compile test If set, CI server will run only non-compile jobs, but no compile jobs or their dependent jobs labels Dec 8, 2022

bors bot merged commit 20ddfb7 into RIOT-OS:master Dec 8, 2022

MrKevinWeiss deleted the pr/dist/compilelikemurdock branch December 12, 2022 10:21

	('meters driven in a Telsa', 5),
	('meters driven in a Tesla', 5),

dist/tools: Add compile_like_murdock #18803

dist/tools: Add compile_like_murdock #18803

Uh oh!

Conversation

MrKevinWeiss commented Oct 26, 2022

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

MrKevinWeiss commented Oct 26, 2022

Uh oh!

MrKevinWeiss commented Oct 26, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MrKevinWeiss commented Nov 1, 2022

Uh oh!

maribu commented Nov 1, 2022

Uh oh!

MrKevinWeiss commented Nov 1, 2022

Uh oh!

maribu commented Nov 1, 2022

Uh oh!

maribu commented Nov 1, 2022

Uh oh!

MrKevinWeiss commented Nov 1, 2022

Uh oh!

MrKevinWeiss commented Nov 2, 2022

Uh oh!

benpicco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

riot-ci commented Dec 8, 2022

Murdock results

Artifacts

Uh oh!

benpicco commented Dec 8, 2022

Uh oh!

bors bot commented Dec 8, 2022

Uh oh!

MrKevinWeiss commented Dec 12, 2022

Uh oh!

MrKevinWeiss commented Dec 12, 2022

Uh oh!

Uh oh!