Inconsistent/undefined behavior between default merges and those with profiles #11091
Description
Description
When passing in -f a merge happens and resources sharing a name are brought together into a single resource. It's not entirely clear at what level profiles affect this, and the behavior between the up and down commands is inconsistent, so I am not even sure I know what I should be logging.
Expected:
Running compose with two files using -f and no profile flag, docker would first filter out the services with a profile and only merge the default services, or error saying that it could not find the service to launch. (depending on when the merge happened)
Actual:
Compose launches the services that are assigned to a profile when the user has not passed in the profile.
Steps To Reproduce
With these config files:
compose.yml
version: "3.7"
services:
mongo:
image: "mongo:4.0.23-xenial"
container_name: "mongo"
ports:
- "27017:27017"
compose-enterprise.yml
version: "3.7"
services:
mongo:
image: "mongo:4.4-focal"
container_name: "mongo"
profiles:
- "cluster"
ports:
- "27017:27017"
restart: always
command: mongos -f /etc/mongos.conf
depends_on:
mongo-shard-1:
condition: "service_started"
mongo-shard-2:
condition: "service_started"
mongo-shard-3:
condition: "service_started"
mongo-shard-1:
image: "mongo:4.4-focal"
container_name: "mongo-shard-1"
profiles:
- "cluster"
ports:
- "27111:27018"
restart: always
command: -f /etc/mongod.conf
mongo-shard-2:
image: "mongo:4.4-focal"
container_name: "mongo-shard-2"
profiles:
- "cluster"
ports:
- "27112:27018"
restart: always
command: -f /etc/mongod.conf
mongo-shard-3:
image: "mongo:4.4-focal"
container_name: "mongo-shard-3"
profiles:
- "cluster"
ports:
- "27113:27018"
restart: always
command: -f /etc/mongod.conf
Running these commands:
When running just the compose.yml file, I get the expected behavior from both up/down
dockertest % docker-compose -f docker/compose.yml up -d mongo
[+] Building 0.0s (0/0) docker:desktop-linux
[+] Running 2/2
✔ Network docker_default Created 0.0s
✔ Container mongo Started 0.0s
dockertest % docker-compose -f docker/compose.yml down mongo
[+] Running 2/2
✔ Container mongo Removed 0.2s
✔ Network docker_default Removed 0.1s
dockertest % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
52cc478bfdd7 mongo:4.0.23-xenial "docker-entrypoint.s…" 9 seconds ago Up 8 seconds 0.0.0.0:27017->27017/tcp mongo
dockertest %
When I run it with the a second -f param file with profiles (compose-extended.yml) however, the behavior is definitely inconsistent:
dockertest % docker-compose -f docker/compose.yml -f docker/compose-enterprise.yml up -d mongo
[+] Building 0.0s (0/0) docker:desktop-linux
[+] Running 5/5
✔ Network docker_default Created 0.0s
✔ Container mongo-shard-3 Started 0.0s
✔ Container mongo-shard-2 Started 0.0s
✔ Container mongo-shard-1 Started 0.0s
✔ Container mongo Started
dockertest % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afc030a766aa mongo:4.4-focal "docker-entrypoint.s…" 9 seconds ago Restarting (2) Less than a second ago mongo
e88ff2f193c6 mongo:4.4-focal "docker-entrypoint.s…" 9 seconds ago Restarting (2) Less than a second ago mongo-shard-3
d5741450e81e mongo:4.4-focal "docker-entrypoint.s…" 9 seconds ago Restarting (2) 1 second ago mongo-shard-1
e36ad3fb436f mongo:4.4-focal "docker-entrypoint.s…" 9 seconds ago Restarting (2) Less than a second ago mongo-shard-2
dockertest %
As you can see - the image is entirely different for the mongo service, so the merge happened - but the profile was not respected on the up command. I would have expected either an unmerged xenial mongo instance stood up (yay!), or an error saying "no such service: mongo" (weird, but.. ok?)
Here's where it gets inconsistent - the down command will actually give you an error!
dockertest % docker-compose -f docker/compose.yml -f docker/compose-enterprise.yml down mongo
no such service: mongo
dockertest %
And if you pass it the profile, it respects it:
dockertest % docker-compose --profile cluster -f docker/compose.yml -f docker/compose-enterprise.yml down mongo
[+] Running 2/0
✔ Container mongo Removed 0.0s
✔ Network docker_default Removed 0.1s
dockertest %
But it also removes the network? Which means a docker ps will not show any running containers... but compose definitely thinks something is running. If you run the same down command without mongo it removes the whole remaining stack:
dockertest % docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dockertest % docker-compose --profile cluster -f docker/compose.yml -f docker/compose-enterprise.yml down
[+] Running 3/0
✔ Container mongo-shard-2 Removed 0.0s
✔ Container mongo-shard-1 Removed 0.0s
✔ Container mongo-shard-3 Removed 0.0s
dockertest %
### Compose Version
```Text
dockertest % docker compose version
Docker Compose version v2.22.0-desktop.2
dockertest % docker-compose --version
Docker Compose version v2.22.0-desktop.2
### Docker Environment
```Text
Client:
Version: 24.0.6
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2-desktop.5
Path: /Users/eric.barry/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.22.0-desktop.2
Path: /Users/eric.barry/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/eric.barry/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.20
Path: /Users/eric.barry/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.8
Path: /Users/eric.barry/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/eric.barry/.docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /Users/eric.barry/.docker/cli-plugins/docker-scan
scout: Docker Scout (Docker Inc.)
Version: v1.0.7
Path: /Users/eric.barry/.docker/cli-plugins/docker-scout
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 24.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.4.16-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 11
Total Memory: 19.52GiB
Name: docker-desktop
ID: 6811903b-982f-487f-91da-31f9e4b83e7a
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
### Anything else?
I don't know if this is a valid use case - but I was looking to create a service that would be overridden when another file was used, but I didn't want that behavior to automatically happen by default. In my example I would expect the "enterprise" file to have more services that could be enabled by default - but the clustering was definitely something you needed to opt-into. If there is a better pattern for this then let me know, but each of the merge patterns I have seen have some drawbacks to them, making file based one the better option.