buildsystem: Remove `buildtest` Goal & Fix `info-buildsizes-diff` #21359

crasbe · 2025-04-04T08:21:53Z

Contribution description

The buildtests goal was an early version used for compiling all boards.

There is still a reference in the documentation that has to be changed, but I'm not familiar with make generate-makefiles.ci, so input in this regard is very welcome.

RIOT/doc/doxygen/src/advanced-build-system-tricks.md

Lines 178 to 196 in fb7acf5

    
           Comparing Build Sizes                                  {#comparing-build-sizes} 
        
           ===================== 
        
           There is a make target for build size comparison. You can use it like that: 
        
           ~~~~~~~~~~~~~~~~~~~ 
        
           $ cd RIOT/test/test_something 
        
           $ git checkout master 
        
           $ BINDIRBASE=master-bin make buildtest 
        
           $ git checkout my-branch 
        
           $ BINDIRBASE=my-branch-bin make buildtest 
        
           $ OLDBIN=master-bin NEWBIN=my-branch-bin make info-buildsizes-diff 
        
           text    data    bss     dec     BOARD/BINDIRBASE 
        
           0       0       0       0       avsextrem    **← this line contains the diff** 
        
           57356   1532    96769   155657  master-bin 
        
           57356   1532    96769   155657  my-branch-bin

Likewise in dist/tools/compile_test/compile_test.py:

RIOT/dist/tools/compile_test/compile_test.py

Lines 127 to 130 in fb7acf5

    
           subprocess = Popen((MAKE, 'buildtest'), 
        
                              bufsize=1, stdin=null, stdout=PIPE, stderr=null, 
        
                              cwd=app_dir, 
        
                              env=subprocess_env)

Testing procedure

Make sure your favorite board and favorite application still build.

Make sure that no references to buildtest or buildtests remain, for example with grep -Rnw . -e buildtest.

Issues/PRs references

Fixes #9742.
Alternative approach for #21355.

crasbe · 2025-04-04T11:11:14Z

The example for info-buildsizes-diff should be changed IMO. Building everything with the amount of boards we have (and the amount of different platforms) will take FOREVER.

Furthermore, info-buildsizes-diff itself is incredibly slow. My proposal would be to check the boards that are actually present in the two folders and just comparing those.

Also, it will show ERR when the build size is equal, which is not really a good experience. There is no comparison to equal to 0.

RIOT/makefiles/info-global.inc.mk

Lines 129 to 130 in fb7acf5

    
           if [[ "$${DIFF}" -gt 0 ]]; then $(COLOR_ECHO) -n "$(COLOR_RED)"; fi; \ 
        
           if [[ "$${DIFF}" -lt 0 ]]; then $(COLOR_ECHO) -n "$(COLOR_GREEN)"; fi; \

The only place where dist/tools/compile_test/compile_test.py is mentioned is here:

RIOT/doc/doxygen/src/getting-started.md

Lines 318 to 331 in fb7acf5

    
           ### Downloading and testing RIOT docker container 
        
           Finally, download the pre-built RIOT Docker container: 
        
           ```console 
        
           # docker pull riot/riotbuild 
        
           ``` 
        
           This will take a while. If it finishes correctly, you can then use the toolchains contained in the Docker container: 
        
           (**from the riot root**): 
        
           ```console 
        
           $ docker run --rm -i -t -u $UID -v $(pwd):/data/riotbuild riot/riotbuild ./dist/tools/compile_test/compile_test.py 
        
           ```

But compile_test.py will build EVERYTHING for ALL boards, which takes forever on a normal machine.
Even generate-Makefile.ci crashed my WSL because it exceeded 15GB of RAM...

Is that used for something else? Or is it even useful to begin with?
The compile_like_murdock.py script in the same folder is used by the GitHub workflows:

RIOT/.github/workflows/tools-test.yml

Lines 41 to 42 in fb7acf5

    
           - name: Test compile_like_murdock script 
        
             run: cd dist/tools/compile_test/tests && ./test.sh

maribu

Thx for cleaning this up!

maribu · 2025-04-04T11:17:14Z

examples/basic/hello-world/README.md

@@ -9,7 +9,6 @@ This example should foremost give you an overview how to use the Makefile system
 * First you must give your application a name, which is commonly the same as the name of the directory it resides in.
  Then you can define a default BOARD for which the application was written.
  By using e.g. `make BOARD=msba2` you can override the default board.


Ah, the good old MSB-A2 still is the "default" board in our doc :D

maribu · 2025-04-04T11:19:13Z

Even generate-Makefile.ci crashed my WSL because it exceeded 15GB of RAM...

And 14.2 GiB of that were the riot/riotbuild docker image :/

crasbe · 2025-04-04T12:12:59Z

It turns out that info-buildsizes-diff does not even work in current master because of a similar issue as described in the other PR. The make system gives the MAKEFLAGS to the submake, which defines BINDIR, which is already defined to native64, since no BOARD is given for the original call of info-buildsizes-diff.

I changed the behavior of info-buildsizes-diff to only show the boards that are actually present in the NEWBIN and OLDBIN and it also compiles a list of boards that don't have a complementary partner in the other directory.
Furthermore if there is no size difference, the number will be printed yellow. Because why not.

Now info-buildsizes-diff is actually useful again. Contrary to info-buildsizes, which I don't think has any real uses tbh.
But I could also add it back in with the same change that I did to info-buildsizes-diff to only evaluate the boards acutally present in the folder.

~~The documentation still has to be adapted.~~ done.

crasbe · 2025-04-04T13:03:56Z

Okay so the dist/tools/ci/build_and_test.sh script became obsolete in #15667, when it was replaced with the static tests as we know them today. This was essentially the last script that called RIOT/dist/tools/compile_test/compile_test.py.

The documentation in

RIOT/doc/doxygen/src/getting-started.md

Lines 318 to 331 in fb7acf5

    
           ### Downloading and testing RIOT docker container 
        
           Finally, download the pre-built RIOT Docker container: 
        
           ```console 
        
           # docker pull riot/riotbuild 
        
           ``` 
        
           This will take a while. If it finishes correctly, you can then use the toolchains contained in the Docker container: 
        
           (**from the riot root**): 
        
           ```console 
        
           $ docker run --rm -i -t -u $UID -v $(pwd):/data/riotbuild riot/riotbuild ./dist/tools/compile_test/compile_test.py 
        
           ```

has to be updated as well, because docker pull riot/riotdocker does not work anymore.

The riotbuild/Dockerfile also mentions this script, but does not execute it:
https://github.com/RIOT-OS/riotdocker/blob/004fdc18820f576267a332def5075b58fbdefc88/riotbuild/Dockerfile#L19-L21

crasbe · 2025-04-04T13:19:36Z

I removed this from the merge queue for now, so someone can take a second look at it again, now that two more scripts are removed and the documentation is updated.

dist/tools/zsh-completion/zsh-riot.sh

doc/doxygen/src/getting-started.md

crasbe · 2025-04-04T15:30:39Z

The makefiles/info-nproc.inc.mk script was only used by the buildtest target and the info-concurrency target was not defined as a global target, therefore not accessible anyway.

Without buildtest, this script is not needed anymore either.

riot-ci · 2025-04-04T17:57:09Z

Murdock results

✔️ PASSED

7543be5 buildsystem: Remove unused scripts and update docker docs

Success	Failures	Total	Runtime
10282	0	10282	09m:47s

Artifacts

Documentation preview

mguetschow · 2025-05-23T12:52:38Z

@crasbe with the riotdocker CI still using buildtest would compile_like_murdock.py be the right tool to switch to? RIOT-OS/riotdocker#256 (comment)

crasbe · 2025-05-23T15:42:13Z

I thought I researched this more thoroughly, but apparently that got missed. Sorry for the mess that it caused.

To answer the question: yes, that should be a valid substitution. However one thing that still has to be checked it how the Github workflow evaluates whether the run was successful or not. As far as I can tell, the Python script always returns success?

See-Also: RIOT-OS/RIOT#21359

crasbe added State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation labels Apr 4, 2025

crasbe requested a review from jia200x as a code owner April 4, 2025 08:21

github-actions bot added Area: doc Area: Documentation Area: build system Area: Build system Area: examples Area: Example Applications labels Apr 4, 2025

crasbe closed this Apr 4, 2025

crasbe reopened this Apr 4, 2025

crasbe mentioned this pull request Apr 4, 2025

buildsystem: Call Buildtest with Clean Environment #21355

Closed

maribu approved these changes Apr 4, 2025

View reviewed changes

github-actions bot added the Area: tools Area: Supplementary tools label Apr 4, 2025

crasbe force-pushed the pr/remove_buildtest branch from 3d1ef72 to c5450fc Compare April 4, 2025 12:14

crasbe added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Apr 4, 2025

crasbe force-pushed the pr/remove_buildtest branch from c5450fc to 1670077 Compare April 4, 2025 12:25

crasbe requested a review from aabadie as a code owner April 4, 2025 12:25

maribu changed the title ~~buildsystem: Remove buildtest Goal~~ buildsystem: Remove buildtest Goal & Fix info-buildsizes-diff Apr 4, 2025

maribu enabled auto-merge April 4, 2025 12:40

crasbe disabled auto-merge April 4, 2025 13:19

mguetschow reviewed Apr 4, 2025

View reviewed changes

dist/tools/zsh-completion/zsh-riot.sh Show resolved Hide resolved

mguetschow reviewed Apr 4, 2025

View reviewed changes

doc/doxygen/src/getting-started.md Outdated Show resolved Hide resolved

crasbe force-pushed the pr/remove_buildtest branch from 3e44741 to 0d15ac7 Compare April 4, 2025 13:27

crasbe removed the State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet label Apr 4, 2025

buildsystem: remove "buildtest" goal

90ea940

crasbe added 2 commits April 5, 2025 21:59

buildsystem: remove info-buildsizes, fix info-buildsizes-diff

612401b

buildsystem: Remove unused scripts and update docker docs

7543be5

crasbe force-pushed the pr/remove_buildtest branch from 9563487 to 7543be5 Compare April 5, 2025 20:00

crasbe enabled auto-merge April 5, 2025 20:04

crasbe added this pull request to the merge queue Apr 5, 2025

Merged via the queue into RIOT-OS:master with commit 6bf28bb Apr 6, 2025
25 checks passed

mguetschow added this to the Release 2025.04 milestone Apr 8, 2025

mguetschow mentioned this pull request May 23, 2025

build.yml: update links to RIOT examples RIOT-OS/riotdocker#256

Merged

chrysn mentioned this pull request Jun 4, 2025

build-system: pin new docker image #21531

Merged

chrysn added a commit to RIOT-OS/rust-riot-sys that referenced this pull request Jun 25, 2025

CI: Use compile_like_murdock instead of buildtest

c8beb8d

See-Also: RIOT-OS/RIOT#21359

chrysn mentioned this pull request Jun 25, 2025

CI: Use compile_like_murdock instead of buildtest RIOT-OS/rust-riot-sys#59

Merged

chrysn added a commit to RIOT-OS/rust-riot-sys that referenced this pull request Jun 25, 2025

CI: Use compile_like_murdock instead of buildtest

a7c9877

See-Also: RIOT-OS/RIOT#21359

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

buildsystem: Remove `buildtest` Goal & Fix `info-buildsizes-diff` #21359

buildsystem: Remove `buildtest` Goal & Fix `info-buildsizes-diff` #21359

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

maribu left a comment

Uh oh!

maribu Apr 4, 2025

Uh oh!

maribu commented Apr 4, 2025

Uh oh!

crasbe commented Apr 4, 2025 •

edited

Loading

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

riot-ci commented Apr 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

mguetschow commented May 23, 2025

Uh oh!

crasbe commented May 23, 2025

Uh oh!

Uh oh!

	Comparing Build Sizes {#comparing-build-sizes}
	=====================
	There is a make target for build size comparison. You can use it like that:

	~~~~~~~~~~~~~~~~~~~
	$ cd RIOT/test/test_something

	$ git checkout master
	$ BINDIRBASE=master-bin make buildtest

	$ git checkout my-branch
	$ BINDIRBASE=my-branch-bin make buildtest

	$ OLDBIN=master-bin NEWBIN=my-branch-bin make info-buildsizes-diff
	text data bss dec BOARD/BINDIRBASE

	0 0 0 0 avsextrem ← this line contains the diff
	57356 1532 96769 155657 master-bin
	57356 1532 96769 155657 my-branch-bin

	subprocess = Popen((MAKE, 'buildtest'),
	bufsize=1, stdin=null, stdout=PIPE, stderr=null,
	cwd=app_dir,
	env=subprocess_env)

buildsystem: Remove buildtest Goal & Fix info-buildsizes-diff #21359

buildsystem: Remove buildtest Goal & Fix info-buildsizes-diff #21359

Uh oh!

Conversation

crasbe commented Apr 4, 2025

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

maribu left a comment

Choose a reason for hiding this comment

Uh oh!

maribu Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

maribu commented Apr 4, 2025

Uh oh!

crasbe commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

crasbe commented Apr 4, 2025

Uh oh!

riot-ci commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Murdock results

Artifacts

Uh oh!

Uh oh!

mguetschow commented May 23, 2025

Uh oh!

crasbe commented May 23, 2025

Uh oh!

Uh oh!

buildsystem: Remove `buildtest` Goal & Fix `info-buildsizes-diff` #21359

buildsystem: Remove `buildtest` Goal & Fix `info-buildsizes-diff` #21359

crasbe commented Apr 4, 2025 •

edited

Loading

riot-ci commented Apr 4, 2025 •

edited

Loading