As Richard Smith points out in https://groups.google.com/a/isocpp.org/d/msg/std-proposals/McSXSFki08I/fwKKfZqNCAAJ, your byte implementation would be unsafe under strict aliasing optimisation.

The solution is to mark up your byte type with __attribute__((__may_alias__)) on GCC and clang, For MSVC, currently, unless you use __declspec(noalias) and/or __declspec(restrict), everything is assumed to alias, though Microsoft promise that strict aliasing optimisation is coming at some point soon.