As per the forum discussion at https://caddy.community/t/reverse-proxy-to-dynamic-ipv4-only/24773/3 , I found it's not possible to disable IPv4 or IPv6 in a reverse_proxy dynamic a block from the Caddyfile.

If I use

:9000 {
	reverse_proxy {
		dynamic a potoroo 8123 {
			versions ipv4
		}
	}
}

then I get a caddy.json which has "versions":{"ipv4":true}}, but Caddy still uses IPv6 if available.

If I explicitly add "ipv6":"false" then it works, but this can't be set from the Caddyfile (that I'm aware of).