Logs stop consuming for container, blocking writes to stderr #6018
Description
Description of the issue
Log output of a postgres container stops after running for a while which is causing subsequent database queries to hang due to failing to write to stderr.
Context information (for bug reports)
Output of "docker-compose version"
$ docker-compose version
docker-compose version 1.21.2, build a133471`
$ python -c 'import docker; print docker.version'
3.3.0
Output of "docker version"
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64
Experimental: false
Output of "docker-compose config"
Here's a snippet of the relevant parts:
networks:
default:
ipam:
config:
- subnet: 192.168.42.0/24
services:
web:
build:
context: /usr/src/app
environment:
POSTGRES_1_PORT: tcp://postgres:5432
expose:
- '8000'
links:
- postgres
ports:
- 8000:8000/tcp
tmpfs:
- /tmp:rw,noexec,nosuid,nodev,size=1g,mode=777
volumes:
- /usr/src/app:/app:rw
- state:/data:rw
- runtime:/run/app:rw
postgres:
environment:
PGDATA: /data/postgres
image: postgres:9.3
volumes:
- state:/data:rw
- runtime:/run/postgresql:rw
version: '2.2'
volumes:
runtime: {}
state: {}
Steps to reproduce the issue
- Run
docker-compose up - Run application test suite from the
webcontainer (this will connect to the postgres container, and many queries executed will cause postgres to write out to stderr)
Observed result
The application test suite will eventually hang. strace on the postgres process running the query in the container shows that it's blocked trying to write to stderr:
12:46:20 write(2, "ERROR: "..., 119
If I run sudo tail -F /proc/<pid>/fd/2 on the same process, the application test suite is no longer hung and resumes.
Expected result
The application test suite should complete without hanging.
Stacktrace / full error message
N/A
Additional information
OS version / distribution, docker-compose install method, etc.
- Ubuntu 16.04 64-bit
pip install docker-compose
The last known good version combination where I cannot reproduce the issue is docker-compose 1.20.1 + docker-py 3.1.4. I start to see the issue if I upgrade docker-py to 3.2.0. A bisect points to this commit as the culprit: docker/docker-py@e9f31e1