Summary

When using the containerd runtime, the containers spawned for jobs do not have the same MTU as the concourse worker container. This means that if the worker has an MTU of lower than 1500 the containers will incorrectly perform TCP MSS, leading to randomly dropped packets and stalled connections.

Steps to reproduce

1. Perform the standard concourse "getting started" setup with the following modifications to docker-compose.yml:

Add the following network to set the worker container's MTU to 1480:

networks:                                
  default:                               
    driver: bridge                       
    driver_opts:                         
      com.docker.network.driver.mtu: 1480

Set the CONCOURSE_RUNTIME environment variable (on the worker) to containerd

2. Apply the following pipeline (this is an example I've found is nearly guaranteed to run into the issue within a few runs, but to be clear the issue effects essentially any pipeline that accesses the internet):

---
resources:
  - name: debian
    type: registry-image
    icon: docker
    check_every: never
    source:
      repository: docker.io/debian
      tag: buster
jobs:
  - name: mtu-example
    plan:
      - get: debian
      - task: example
        image: debian
        config:
          platform: linux
          run:
            path: /bin/sh
            args:
              - -exc
              - |
                apt-get update -y
                apt-get install -y --no-install-recommends sl

3. Unpause the pipeline and start the build. It may take a few builds, but apt should fail to update eventually. A more precise way of detecting the issue is shown below in Expected/Actual results.

Example of one of my failures testing this:

E: Failed to fetch http://deb.debian.org/debian/pool/main/n/ncurses/libncurses6_6.1+20181013-2+deb10u2_amd64.deb  Connection failed [IP: 151.101.6.132 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Expected results

The job containers should have the same MTU as the worker container. Alternatively, concourse should offer an option to configure the MTU for containerd.

Actual results

First, look at the result from docker execing into the concourse container:

$ docker exec -it concourse_worker_1 bash           
root@6ce4541bb038:/src# ip link show eth0
26: eth0@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default 
    link/ether 02:42:ac:13:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.19.0.4/16 brd 172.19.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Note the interface has mtu 1480.

Now, hijack the container from the build:

$ fly -t dev hijack -j example/mtu-example
root@9f7acdff-01d8-4acf-75b3-a7864737234c:/tmp/build/c3499c27# ip link show eth0
3: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c6:e4:dc:fe:d2:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.80.0.12/16 brd 10.80.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c4e4:dcff:fefe:d263/64 scope link 
       valid_lft forever preferred_lft forever

Note the interface has mtu 1500.

Additional context

My production cluster's containers have an MTU of 1480 because of the tunneling used by the kubernetes CNI plugin I am using. I suspect many kubernetes clusters running concourse will run into this hard to debug issue.

Triaging info

• Concourse version: 7.0.0
• Browser (if applicable): N/A
• Did this used to work? As far as I know, no