I'm writing/written a plugin for Caddy which watches a Nomad / Consul cluster and builds ingress routes to our cluster based on the data, basically it generates a collection of reverse_proxy statements to the running containers.

However I've noticed that after reloading the Caddyfile existing clients continue to use the previous configurations for 2 - 3 minutes or until the browser cache is cleared.

Initially I thought this was something I'd done wrong in my code, so I've extracted the minimum Caddyfile and manually applied it against a stock version of Caddy (2.7.6 running in Docker) with the same results.

Initially I might have a reverse_proxy block such as

reverse_proxy {
      dynamic srv {
	name php-pool-1.service.consul
        refresh 10s
        dial_timeout 5s
      }
      import reverseProxyConfig
      transport http {
        keepalive off
      }
    }

After a deployment the php-pool-1 name changes e.g.

reverse_proxy {
      dynamic srv {
	name php-pool-2.service.consul
        refresh 10s
        dial_timeout 5s
      }
      import reverseProxyConfig
      transport http {
        keepalive off
      }
    }

If I start browsing the site I can see Caddy logging that it's querying DNS for php-pool-1.service.consul, which is correct.

If I now reload with a new Caddyfile to change the upstream to php-pool-2.service.consul then I can see Caddy logging that it's querying DNS for php-pool-1.service.consul and not the new php-pool-2.service.consul, it will keep using the old upstream until either I clear the browser cache or wait 2 - 3 minutes, after which it moves to php-pool-2.service.consul.

If I use a different browser to connect directly after the reload then that connection is using php-pool-2.service.consul, which is what I expect.

The minimal Caddyfile I have is:

{
  admin localhost:2019

  log {
    output stdout
    level DEBUG
    format console
  }

  grace_period 3s
  debug
}

(reverseProxyConfig) {
  header_up +X_FORWARDED_PORT 443
  lb_policy least_conn
  lb_try_duration 5s
  lb_try_interval 250ms
  fail_duration 2s
  unhealthy_status 5xx
}

(logsConfig) {
  log {
    output stdout
    level DEBUG
    format console
  }
}

# Health Check
:8080 {
  respond /health-check OK 200
}

*.fortix.systems {
  import logsConfig
  encode zstd gzip

  @wildcard_86 host ingresstest.fortix.systems
  handle @wildcard_86 {
    reverse_proxy {
      dynamic srv {
	name php-pool-46db85dd489abc89543ff3ff288c7975.service.consul
        refresh 10s
        dial_timeout 5s
      }
      import reverseProxyConfig
      transport http {
        keepalive off
      }
    }
  }

  handle {
    abort
  }
}

From the documentation, https://caddyserver.com/docs/caddyfile/options#grace-period if I include grace_period then it should force connection close after in my case 3s, but it doesn't seem to be happening so I'm guessing I'm missing something obvious here.

Thank you for any guidance