Buildah reuse gzip layers when "--compression-format zstd" assigned #4613
Description
Description
I pushed my image with flag "--compression-format zstd", but when I checked its manifest I found that it just reused gzip layers.
Steps to reproduce the issue:
- Install buildah from scratch:
sudo mkdir ~/buildah
cd ~/buildah
export GOPATH=`pwd`
sudo git clone https://github.com/containers/buildah ./src/github.com/containers/buildah
cd ./src/github.com/containers/buildah
sudo make BUILDTAGS=containers_image_openpgp
sudo make install
sudo cp /usr/local/bin/buildah /usr/bin/
- Build image:
sudo buildah bud -t placeboy/buildah-from-dockerfile .
Here is my Dockerfile:
FROM node:10
WORKDIR /usr/src/app
RUN npm init -y
RUN npm install express --save
COPY HelloWorld.js .
CMD [ "node", "HelloWorld.js" ]- Push with no compression-format assigned:
sudo buildah push placeboy/buildah-from-dockerfile
- Check manifest:
$ sudo docker buildx imagetools inspect --raw placeboy/buildah-from-dockerfile | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:0ee844816c31997f60ad0c6eeab16094d044b5c2e06319bfbf6d845b88626563",
"size": 7735
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:76b8ef87096fa726adbe8f073ef69bb5664bac19474c5cce4dd69e08a234903b",
"size": 45380037
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:2e2bafe8a0f40509cc10249087268e66a662e437f10e9598a09abb5687038a57",
"size": 11286411
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:b53ce1fd2746e8d2037f1b0b91ddea0cc7411eb3e5949fe10c0320aca8f7392b",
"size": 4342420
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:84a8c1bd5887cc4a89e1f286fed9ee31ce12dba9f6813cf14082ada3e9ab6f63",
"size": 49786874
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:7a803dc0b40fcd10faee3fb3ebb2d7aaa88500520e6295295f5163c4bb48548b",
"size": 214347112
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:b800e94e7303e276b8fb4911a40bfe28f46180d997022c69bf1ee02fb7b86721",
"size": 4190
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:0da9fbf60d485c74d153bfc7562f34533550e3dd8aa78f5e8c2476ed500e0e73",
"size": 21417193
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:04dccde934cf0d6ab9d303fe73b36bc4ed84329d9d562049a69a5d38afb83f14",
"size": 2484048
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:73269890f6fdf96184d0fdb0afe3cf697b0c8dbe1e7a077f0fab445d74372c78",
"size": 294
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:d04736ed33aa34ba551e9525a9b3c3527ed683afa212a39c581c3f5aa7231a00",
"size": 1891961
}
],
"annotations": {
"org.opencontainers.image.base.digest": "sha256:686e0e859358f28bfe1641e1627549b9cd0ad74f222b953d74209213488c6858",
"org.opencontainers.image.base.name": "docker.io/library/node:10"
}
}
- Push the image again, with zstd compression-format assigned:
sudo buildah push --compression-format zstd placeboy/buildah-from-dockerfile
Check manifest again now:
$ sudo docker buildx imagetools inspect --raw placeboy/buildah-from-dockerfile | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:0ee844816c31997f60ad0c6eeab16094d044b5c2e06319bfbf6d845b88626563",
"size": 7735
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:76b8ef87096fa726adbe8f073ef69bb5664bac19474c5cce4dd69e08a234903b",
"size": 45380037
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:2e2bafe8a0f40509cc10249087268e66a662e437f10e9598a09abb5687038a57",
"size": 11286411
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:b53ce1fd2746e8d2037f1b0b91ddea0cc7411eb3e5949fe10c0320aca8f7392b",
"size": 4342420
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:84a8c1bd5887cc4a89e1f286fed9ee31ce12dba9f6813cf14082ada3e9ab6f63",
"size": 49786874
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:7a803dc0b40fcd10faee3fb3ebb2d7aaa88500520e6295295f5163c4bb48548b",
"size": 214347112
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:b800e94e7303e276b8fb4911a40bfe28f46180d997022c69bf1ee02fb7b86721",
"size": 4190
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:0da9fbf60d485c74d153bfc7562f34533550e3dd8aa78f5e8c2476ed500e0e73",
"size": 21417193
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:04dccde934cf0d6ab9d303fe73b36bc4ed84329d9d562049a69a5d38afb83f14",
"size": 2484048
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:73269890f6fdf96184d0fdb0afe3cf697b0c8dbe1e7a077f0fab445d74372c78",
"size": 294
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:d04736ed33aa34ba551e9525a9b3c3527ed683afa212a39c581c3f5aa7231a00",
"size": 1891961
}
],
"annotations": {
"org.opencontainers.image.base.digest": "sha256:686e0e859358f28bfe1641e1627549b9cd0ad74f222b953d74209213488c6858",
"org.opencontainers.image.base.name": "docker.io/library/node:10"
}
}
Describe the results you expected:
$ sudo docker buildx imagetools inspect --raw placeboy/buildah-from-dockerfile | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:29fc3b654b32f36604ec1d4473ed98635e43fae2bd9137392b4097a15c40ce49",
"size": 7734
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:342dd87184b7f48dba284f855fd1acb181f9822a205840066597760040b3c39d",
"size": 36063577
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:3043dc57d5854957650610db9881dedde2b563794c8440a4b0313a78acb7a294",
"size": 9726868
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:17a80d72fc5e7631a931da627fe82cf316246180a38f61ef7e8ac2c709929250",
"size": 3239525
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:dc063313fc998715c944a9216e0fc96def311a2484d593b174cf0bce3635e7b2",
"size": 38695908
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:ec271c8a034a45dc4a8d0c0c415b6fc41ed808762314ea51ea21ff0dbe36ab68",
"size": 183371283
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:c659975e02f6477ae0e36f613b6baecc5687a6271d5e08d834cc8ba47d7188f3",
"size": 3647
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:6427d59e665293132649d4b660e6b59611339ac7d3162ee7f4a2611601583ddf",
"size": 19422790
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:3188faec5ac3e6d2c329d58a4b409cba0dadf2cc16d28381292d42faf92f1e03",
"size": 2450837
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:f8467ea4a05e4636f7163d7f55e009c829ac6913af95b8589d7512030443694c",
"size": 284
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
"digest": "sha256:dcc2ac1ec89846d6267cb56b94235e5e6539084b1c848e13027b31397d4ae16e",
"size": 1792789
}
],
"annotations": {
"org.opencontainers.image.base.digest": "sha256:686e0e859358f28bfe1641e1627549b9cd0ad74f222b953d74209213488c6858",
"org.opencontainers.image.base.name": "docker.io/library/node:10"
}
}
Output of rpm -q buildah or apt list buildah:
package buildah is not installed
Output of buildah version:
WARN[0000] Failed to decode the keys ["storage.options.override_kernel_check"] from "/etc/containers/storage.conf"
Version: 1.30.0-dev
Go Version: go1.20
Image Spec: 1.0.2-dev
Runtime Spec: 1.0.2-dev
CNI Spec: 1.0.0
libcni Version: v1.1.2
image Version: 5.24.1
Git Commit: 5a63d7f61903b1da22343bc021a7f7d64c1c2895
Built: Tue Feb 21 15:03:18 2023
OS/Arch: linux/amd64
BuildPlatform: linux/amd64
Output of cat /etc/*release:
CentOS Linux release 7.8.2003 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.8.2003 (Core)
CentOS Linux release 7.8.2003 (Core)
Output of uname -a:
Linux bjrz-c497.lf 4.18.0-2.6.4.2.kwai.x86_64 #1 SMP Tue Oct 25 12:40:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Output of cat /etc/containers/storage.conf:
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]
# Default Storage Driver
driver = "overlay"
# Temporary storage location
runroot = "/var/run/containers/storage"
# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"
[storage.options]
# Storage options to be passed to underlying storage drivers
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]
# Size is used to set a maximum size of the container image. Only supported by
# certain container storage drivers.
size = ""
# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"
# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs. Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536
# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file. Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"
[storage.options.thinpool]
# Storage Options for thinpool
# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"
# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"
# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"
# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"
# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""
# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"
# fs specifies the filesystem type to use for the base device.
# fs="xfs"
# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"
# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"
# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""
# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""
# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"
# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"
# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"