The reason this is not done is because at some point it was decided that labels are for higher-level tools (that is Docker API clients) and not for passing down configuration.
In such a case it would be docker that sets the annotation that the OCI runtime takes advantage of rather than a user directly setting it.

This of course all happened before OCI and annotations, etc.
containerd now uses labels heavily for configuration (as does k8s).... so maybe it's time to re-evaluate.

Does it make sense to pass down all labels? Can this cause some unexpected behaviors?

I think in previous discussions I was also wondering if Docker labels should be namespaced / prefixed to distinguish them from labels set at lower levels (e.g. com.docker.label.<name of label>=<value of label>)

Implemented (but slightly different) in #45025