"failed to copy: io: read/write on closed pipe" on ctr images push while pushing large images #7972
Description
Description
While pushing large container image to a registry using
ctr images push ghcr.io/akhilerm/private-testing:ci container-registry.oracle.com/database/express:21.3.0-xe
hitting the following error
failed to copy: io: read/write on closed pipe
Notes:
- The issue occurs randomly, so its not easily reproducible. Out of 10 tries, hit the issue only twice.
- Faced while uploading to
ghcr.ioonly. Couldnt reproduce for dockerhub or quay. (Again this can be due to randomness) - Similar issue has been reported in unable to push images to harbor registry moby/buildkit#3347 and buildx failed with: ERROR: failed to solve: failed to push ghcr.io/finchsec/kali:latest: failed to copy: io: read/write on closed pipe docker/build-push-action#761
Steps to reproduce the issue
sudo ctr content fetch --all-platforms container-registry.oracle.com/database/express:21.3.0-xectr images push ghcr.io/akhilerm/testing-gha:io-failure container-registry.oracle.com/database/express:21.3.0-xe
$ ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
container-registry.oracle.com/database/express:21.3.0-xe application/vnd.docker.distribution.manifest.v2+json sha256:016d1a2becd9c9b9bfb683eebf3aa092527fe1354ace5b23691e75759f301bed 3.3 GiB linux/amd64 -Reproduced using this image which is 3.3GiB in size.
I will update more info into the issue, if this can be reproduced easily; as currently testing requires uploading the 3GiB image and can take a lot of time.
Describe the results you received and expected
Expected the image to be successfully pushed to the registry
What version of containerd are you using?
Any other relevant information
$ runc --version
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d
spec: 1.0.2-dev
go: go1.18.9
libseccomp: 2.5.3
$ uname -a
Linux ams-hz-ubu-055 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/LinuxShow configuration if it is related to CRI plugin.
$ cat /etc/containerd/config.toml
# Copyright 2018-2022 Docker Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#disabled_plugins = ["cri"]
#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0
#[grpc]
# address = "/run/containerd/containerd.sock"
# uid = 0
# gid = 0
#[debug]
# address = "/run/containerd/debug.sock"
# uid = 0
# gid = 0
# level = "info"