I'd like to use caddy in a socket-activated environments, using FDs passed down from the service manager, rather than binding on addresses on its own.

Combined with signalling readyness (which caddy already does), this will give zero-downtime (re)deployments on Linux systems using systemd (if .socket files are used), by simply restarting the process - the socket is held open by systemd, and new connections are passed in once caddy is ready to accept new requests. In these cases, there wouldn't be a need for complicated reload logic anymore.

github.com/coreos/go-systemd/activation provides the necessary methods to check whether FDs are passed, including identifying them by their socket name. https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket-activation gives a nice introduction into the feature itself.

In case no explicit listen addresses are specified, caddy could default to do that rather than binding on its own, if it detects it's running in such an environment.
Additionally, Caddyfile could be extended to allow specifying these passed fds as network addresses (something like sd-listen:$name or sd-listen:$idx maybe). This can become useful when you want to expose different things on different sockets.