Expected behavior

I have 3 minio servers in different locations. The first server should replicate to the second server and the second server to the third server. This is done via mc mirror --watch test-2 test-1 and mc mirror --watch test-3 test-2. It should mirror all PUT and DELETE requests from the test-3 minio up to the test-1 minio.

Use case:
The "source" minio server following named test-3 is located in EU and the other two minio servers are located in the USA.
To reduce traffic between USA and EU I want the test-2 minio server to be the source of the test-1 minio server.

Actual behavior

PUT request performed to the test-3 minio only gets mirrored to the test-2 minio and not up the test-1 minio.
On the other hand DELETE requests are performed correctly.

Steps to reproduce the behavior

1. Create 3 Minio Servers:

root@minio-test-1:~# docker create --name test-minio --network host -e MINIO_ROOT_USER=root-0423 -e MINIO_ROOT_PASSWORD=xyz -e MINIO_PROMETHEUS_AUTH_TYPE=public -e MINIO_BROWSER=off  minio/minio server --address 10.0.0.2:9000 /data
root@minio-test-2:~# docker create --name test-minio --network host -e MINIO_ROOT_USER=root-0423 -e MINIO_ROOT_PASSWORD=xyz -e MINIO_PROMETHEUS_AUTH_TYPE=public -e MINIO_BROWSER=off  minio/minio server --address 10.0.0.3:9000 /data
root@minio-test-3:~# docker create --name test-minio --network host -e MINIO_ROOT_USER=root-0423 -e MINIO_ROOT_PASSWORD=xyz -e MINIO_PROMETHEUS_AUTH_TYPE=public -e MINIO_BROWSER=off  minio/minio server --address 10.0.0.4:9000 /data

2. Create a minio config

{
    "aliases": {
        "test-1": {
            "accessKey": "root-0423",
            "api": "s3v4",
            "path": "on",
            "secretKey": "xyz",
            "url": "http://10.0.0.2:9000"
        },
        "test-2": {
            "accessKey": "root-0423",
            "api": "s3v4",
            "path": "on",
            "secretKey": "xyz",
            "url": "http://10.0.0.3:9000"
        },
        "test-3": {
            "accessKey": "root-0423",
            "api": "s3v4",
            "path": "on",
            "secretKey": "xyz",
            "url": "http://10.0.0.4:9000"
        }
    },
    "version": "10"
}

3. Create the mirror containers

root@minio-test-1:~# docker create --name minio-mirror -v $PWD/.mc/:/root/.mc/ --network host minio/mc:latest mirror --json --watch --overwrite test-2 test-1
root@minio-test-2:~# docker create --name minio-mirror -v $PWD/.mc/:/root/.mc/ --network host minio/mc:latest mirror --json --watch --overwrite test-3 test-2

4. Create test bucket on minio test-3
5. Copy a file to minio test-3

You can now perform mc ls test-2/test and mc ls test-1/test. The object will be listed on test-2 but not on test-1.
If you copy the file to test-1 by hand and the delete the object on test-3 all servers will be empty.

mc --version

mc version RELEASE.2023-04-12T02-21-51Z (commit-id=1843717c57fb87612469b7610344a7d49d97a497)
Runtime: go1.20.3 linux/amd64
Copyright (c) 2015-2023 MinIO, Inc.
License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

minio/mc Docker Image