We emphasize containers, but there are needs to execute code on the host. Package layering is one; it has some advantages and major disadvantages.

In OpenShift today, we have a pattern of privileged containers that lay down and execute some binaries on the host. This avoids the reboots inherent in layering (also doesn't require RPMs).

Recently however, this failed because running binaries targeting e.g. RHEL7 on a RHEL8 host may fail if they link to e.g. openssl. A general best practice here is really that the binaries need to be targeting the same userspace as the host. With e.g. statically linked Go/Rust type code one can avoid most issues but not all (and you really want to dynamically link openssl).

See this Dockerfile which pairs with this PR.

Further, I think we should move these binaries into e.g. /run/bin or so - if there's a higher level process that pulls the containers to a node on reboot (e.g. a systemd unit created via Ignition, or in the OpenShift case the kubelet), then having the binaries in /run helps ensure that if e.g. the container is removed, at least the binaries will go away on reboot.

That said...we may consider even shipping something like a /host/usr/bin/coreos-host-overlay install <name> <rootfs> tool that assumes the host rootfs is mounted at /host and handles the case where e.g. a container delivering host binaries is upgraded (or removed) before reboot.

(This problem set quickly generalizes of course to something like "transient RPMs")