Wouldn't it be possible to do this with a mutex so we don't require the thread flags feature?

I think not without going into the guts of the message receive code. thread flags already have special handling in there, that's why I've re-used it.

I wonder how safe is it to assume that the timeout thread flag is not used by any other callback/interrupt?

An alternative to this solution could be adding a mutex that is (try_)locked either from _xtimer_msg_receive_timeout or the timer callback.
The callback would then only send the timeout message if it was able to lock the mutex.
The receiving function then still receives either the timeout message or the proper message and can then decide if it needs to cancel the timer or throw away the unneeded timeout message.
What do you think?

Wouldn't it be possible to do this with a mutex so we don't require the thread flags feature?

It would then require the mutex feature. That isn't optional, but still quite large and not necesarily used (linked in).

I wonder how safe is it to assume that the timeout thread flag is not used by any other callback/interrupt?

That is actually a feature, and a matter of documentation.
Currently, in RIOT, only the xtimer function sets that timeout.
That can be used to set up higher-level timeouts, e.g.,

xtimer_set_timeout_flag(10s);
// dop stuff
xtimer_msg_receive_timeout(foo, 5s);
if timeout: abort
xtimer_msg_receive_timeout(foo, 5s);
if timeout: abort
xtimer_msg_receive_timeout(foo, 5s);
if timeout: abort
// ...

Now the xtimer_msg_receive calls can each take up to 5s, but with the timeout flag based implementation, the whole block cannot take longer than 10s.

But yeah, for that to work the timeout flag shouldn't be reset in xtimer_set_timeout_flag().

How to continue with this one?
Upside: simple solution, no additional mutex handling or other fancy mechanisms needed.
Downside: if any part of the application calls xtimer_set_timeout_flag in a way that overlaps with xtimer_msg_receive_timeout this will cause trouble.
It would be hard (impossible?) to cover just by documentation: yes we can add a statement to both functions, explaining the constraints. But how is the user supposed to know if a module uses xtimer_set_timeout_flag internally?

This solution is better than what we have now but I have the strong feeling someone will be bitten by this again If we try to "fix" the corner cases by documentation.
@kaspar030 what do you think?

This is also pretty old, is it still a problem now?

closing as xtimer is deprecated