Description

When building using buildtest the build is not using the BOARD bin/$(BOARD) directory but the default BOARD defined in the application Makefile BOARD ?= native.

This also has consequences on dist/tools/compile_test/compile_test.py which is using buildtest.

EDIT: This only happens when setting variables on the command line as then the MAKEOVERRIDES from here are done even if these variables are not set.

I think a solution is to only override variables which were set from the command line (by checking origin) before changing there value.

A better fix would be to never override but require variables to have been set absolute if set from the command line, (and maybe also environment) but it changes the behavior.

Steps to reproduce the issue

I am using #9741 to make it more visible during compilation but can be done without.

When building for iotlab-m3 the output is saved in bin/native.

$ git clean -xdff examples/hello-world

$ BOARDS=iotlab-m3 make -C examples/hello-world/ buildtest 'BUILDTEST_MAKE_REDIRECT='
...
Building application "hello-world" for "iotlab-m3" with MCU "stm32f1".

   text    data     bss     dec     hex filename
   8480     140    2740   11360    2c60 /home/harter/work/git/RIOT/examples/hello-world/bin/native/hello-world.elf

$ find examples/hello-world/bin/
examples/hello-world/bin/
examples/hello-world/bin/native
examples/hello-world/bin/native/hello-world.hex
examples/hello-world/bin/native/hello-world.elf
examples/hello-world/bin/native/hello-world.map

When running some compilations with buildtest I also sometime had strange issues with packages with native but was correct when build alone.
I think it is related to an issue with the package clean which was shown with this path collision.

Expected results

Building in bin/iotlab-m3

Actual results

Build in bin/native.

Versions

Ubuntu 16.04