Move remaining image packages to daemon #50446

dmcgowan · 2025-07-17T23:22:36Z

Moves packages associated with imagesand layers to daemon. Moves the relevant internal and pkgs into daemon as well.

thaJeztah

This needs more work; it will break our cli

thaJeztah

Made a first pass, but may need to look at some in more depth

thaJeztah · 2025-07-21T10:18:46Z

daemon/internal/image/spec/README.md

image/spec was kept because it previously defined the image specification; we should consider removing it instead of moving. It has no place as an internal package.

thaJeztah · 2025-07-21T10:21:18Z

daemon/internal/image/v1/imagev1.go

+	"github.com/docker/docker/daemon/internal/image"
 	"github.com/docker/docker/daemon/internal/layer"
-	"github.com/docker/docker/image"


Do we still need this package at all? I started looking at this, but didn't finish; it looks to be used for "pre-1.9 compatibility";

moby/image/tarexport/save.go

Lines 438 to 457 in b10cbd9

v1ImgCreated := time.Unix(0, 0)

v1Img := image.V1Image{

// This is for backward compatibility used for

// pre v1.9 docker.

Created: &v1ImgCreated,

}

if i == len(img.RootFS.DiffIDs)-1 {

v1Img = img.V1Image

}

rootFS := *img.RootFS

rootFS.DiffIDs = rootFS.DiffIDs[:i+1]

v1ID, err := v1.CreateID(v1Img, rootFS.ChainID(), parent)

if err != nil {

return nil, err

}

v1Img.ID = v1ID.Encoded()

if parent != "" {

v1Img.Parent = parent.Encoded()

}

daemon/internal/distribution/pull.go

daemon/command/config.go

thaJeztah · 2025-07-21T10:46:33Z

daemon/dockerversion/version_lib.go

This one will require changes to our build-scripts;

moby/hack/make/.go-autogen

Lines 3 to 9 in b10cbd9

LDFLAGS="${LDFLAGS} \

-X \"github.com/docker/docker/dockerversion.Version=${VERSION}\" \

-X \"github.com/docker/docker/dockerversion.GitCommit=${GITCOMMIT}\" \

-X \"github.com/docker/docker/dockerversion.BuildTime=${BUILDTIME}\" \

-X \"github.com/docker/docker/dockerversion.PlatformName=${PLATFORM}\" \

-X \"github.com/docker/docker/dockerversion.ProductName=${PRODUCT}\" \

-X \"github.com/docker/docker/dockerversion.DefaultProductLicense=${DEFAULT_PRODUCT_LICENSE}\" "

moby/hack/make.ps1

Lines 487 to 492 in b10cbd9

$ldflags = "-X 'github.com/docker/docker/dockerversion.Version="+$dockerVersion+"'"

$ldflags += " -X 'github.com/docker/docker/dockerversion.GitCommit="+$gitCommit+"'"

$ldflags += " -X 'github.com/docker/docker/dockerversion.BuildTime="+$env:BUILDTIME+"'"

$ldflags += " -X 'github.com/docker/docker/dockerversion.PlatformName="+$env:PLATFORM+"'"

$ldflags += " -X 'github.com/docker/docker/dockerversion.ProductName="+$env:PRODUCT+"'"

$ldflags += " -X 'github.com/docker/docker/dockerversion.DefaultProductLicense="+$env:DEFAULT_PRODUCT_LICENSE+"'"

daemon/pkg/plugins/client.go

daemon/pkg/stringid/stringid.go

daemon/pkg/system/args_windows.go

thaJeztah · 2025-07-21T12:16:10Z

daemon/pkg/tarsum/tarsum_spec.md

pkg/tarsum also seems to define the specification; is it something to be preserved and moved separate from this repo?

I think buildkit should own it, we have no use for it in the future in the engine but it is still useful for build

thaJeztah · 2025-07-21T12:17:30Z

daemon/pkg/tarsum/tarsum.go

It looks like the only remaining use of this is in builder/remotecontext` (for the classic builder); do we know if it's still needed there? How does BuildKit do this? (ISTR it had its own implementation, which made me curious if we can share that if it's still needed); I'm not very familiar with how/why it's used exactly;

moby/daemon/builder/remotecontext/filehash.go

Lines 30 to 45 in a1ee566

tsh := &tarsumHash{hdr: hdr, Hash: sha256.New()}

tsh.Reset() // initialize header

return tsh, nil

}

type tarsumHash struct {

hash.Hash

hdr *tar.Header

}

// Reset resets the Hash to its initial state.

func (tsh *tarsumHash) Reset() {

// comply with hash.Hash and reset to the state hash had before any writes

tsh.Hash.Reset()

tarsum.WriteV1Header(tsh.hdr, tsh.Hash)

}

vvoland

SGTM

Signed-off-by: Derek McGowan <derek@mcg.dev>

…networking Signed-off-by: Derek McGowan <derek@mcg.dev>

Signed-off-by: Derek McGowan <derek@mcg.dev>

austinvazquez

Been trending positive on this but giving it the official thumbsup. My mindset is we can continue to rehome inside the daemon module once it is created before stabilizing the module and creating a tag there.

dmcgowan · 2025-07-25T00:54:54Z

@austinvazquez I think the general approach we agreed on earlier is to at least put all the daemon stuff that we know is not used in daemon. The other packages we know are being legitimately used, we can keep them outside. That gives us the option of ensuring that nothing under daemon is imported anywhere.

austinvazquez · 2025-07-25T01:31:13Z

@dmcgowan, okay most of this move still makes sense to me. The one I'm having trouble now with is pkg/tarsum. Would that stay in place?

dmcgowan · 2025-07-25T04:24:21Z

tarsum isn't used by any active projects, only used internally. There is probably more we can move to internal after a bit more vetting/splitting.

thaJeztah

LGTM, thanks!

thaJeztah · 2025-07-25T10:33:12Z

daemon/containerfs_linux.go

-	"github.com/docker/docker/internal/unshare"
+	"github.com/docker/docker/daemon/internal/unshare"


Not for this PR; considering if this package could be moved to a module in moby/sys; it looks very self-contained and stable, and we currently maintain a fork of the code in moby/go-archive/chrootarchive.

thaJeztah · 2025-07-25T10:39:12Z

tarsum isn't used by any active projects, only used internally. There is probably more we can move to internal after a bit more vetting/splitting.

Yes, there's a bunch of things that need curating. Split cruft from relevant parts.

thaJeztah · 2025-07-25T11:03:32Z

Double-checked some of my older comments; I left some of them "unresolved" as a reminder for some follow-ups. I just merged my PR in docker/cli to remove the registry dependency there, which paves the way for more cleaning up;

add internal fork of docker/docker/registry docker/cli#6207

Let's bring this one in 👍 🎉

dmcgowan requested a review from samuelkarp as a code owner July 17, 2025 23:22

dmcgowan mentioned this pull request Jul 17, 2025

Create separate go modules for api, client, and daemon #49873

Closed

17 tasks

dmcgowan added this to go modules transition Jul 17, 2025

dmcgowan moved this to In Progress in go modules transition Jul 17, 2025

dmcgowan added this to the 29.0.0 milestone Jul 17, 2025

dmcgowan added the release-blocker PRs we want to block a release on label Jul 17, 2025

dmcgowan force-pushed the move-image-daemon branch from f5597ad to 75231e8 Compare July 18, 2025 00:03

dmcgowan requested a review from tianon as a code owner July 18, 2025 00:03

dmcgowan force-pushed the move-image-daemon branch from 75231e8 to 02bd9a3 Compare July 18, 2025 00:18

dmcgowan self-assigned this Jul 18, 2025

thaJeztah requested changes Jul 18, 2025

View reviewed changes

thaJeztah mentioned this pull request Jul 21, 2025

remove uses of lazyregexp in tests, test-utilities and packages used externally #50465

Merged

thaJeztah reviewed Jul 21, 2025

View reviewed changes

thaJeztah mentioned this pull request Jul 21, 2025

driver/kubernetes: remove uses of pkg/homedir docker/buildx#3324

Merged

dmcgowan mentioned this pull request Jul 14, 2025

Support for external daemon package importers #50372

Open

24 tasks

dmcgowan force-pushed the move-image-daemon branch 4 times, most recently from 61a4135 to 3d4e2ca Compare July 24, 2025 04:53

vvoland approved these changes Jul 24, 2025

View reviewed changes

thompson-shaun mentioned this pull request Jul 24, 2025

Create github.com/moby/moby/daemon module #50430

Closed

dmcgowan added 9 commits July 24, 2025 12:10

Move layer to daemon/internal/layer

fa9a3c3

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move distribution to daemon/internal/distribution

1ff6011

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move image to daemon/internal/image

f24455c

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move oci to daemon/pkg/oci

c74ba95

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move reference to daemon/internal/refstore

f1f83da

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move registry to daemon/pkg/registry

7bfb804

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move pkg/idtools to daemon/internal/idtools

ca5d234

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move pkg/tarsum to daemon/builder/remotecontext/internal/tarsum

9566272

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/lazyregexp to daemon/internal/lazyregexp

222b2b8

Signed-off-by: Derek McGowan <derek@mcg.dev>

dmcgowan added 14 commits July 24, 2025 12:12

Move internal/cleanups to daemon/internal/cleanups

ca9c4df

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/containerfs to daemon/internal/containerfs

50743e3

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/ioutils to daemon/internal/ioutils

baea1c5

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/multierror to daemon/internal/multierror

fd21e3b

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/nlwrap to daemon/libnetwork/nlwrap

713d7f5

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/opts to daemon/internal/opts

b62b542

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/otelutil to daemon/internal/otelutil

c3b0e01

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/platform to daemon/internal/platform

8042010

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/rootless to daemon/internal/rootless

20cc1a6

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/sliceutil to daemon/internal/sliceutil

5b913b3

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/unix_noeintr to daemon/internal/unix_noeintr

28de87f

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/unshare to daemon/internal/unshare

14eb277

Signed-off-by: Derek McGowan <derek@mcg.dev>

Move internal/testutils/networking to integration/internal/testutils/…

6514282

…networking Signed-off-by: Derek McGowan <derek@mcg.dev>

Update golangci for lazyregexp

0abcdb7

Signed-off-by: Derek McGowan <derek@mcg.dev>

dmcgowan force-pushed the move-image-daemon branch from 3d4e2ca to 0abcdb7 Compare July 24, 2025 19:21

dmcgowan changed the title ~~Move remaining daemon packages~~ Move remaining image packages to daemon Jul 24, 2025

austinvazquez approved these changes Jul 25, 2025

View reviewed changes

thaJeztah approved these changes Jul 25, 2025

View reviewed changes

thaJeztah merged commit 57e1cb2 into moby:master Jul 25, 2025
213 of 215 checks passed

github-project-automation bot moved this from In Progress to Done in go modules transition Jul 25, 2025

This was referenced Jul 31, 2025

pkg/compose: simplify getting auth-config key docker/compose#13120

Merged

daemon/internal/image: remove image-spec markdown #50605

Merged

	v1ImgCreated := time.Unix(0, 0)
	v1Img := image.V1Image{
	// This is for backward compatibility used for
	// pre v1.9 docker.
	Created: &v1ImgCreated,
	}
	if i == len(img.RootFS.DiffIDs)-1 {
	v1Img = img.V1Image
	}
	rootFS := *img.RootFS
	rootFS.DiffIDs = rootFS.DiffIDs[:i+1]
	v1ID, err := v1.CreateID(v1Img, rootFS.ChainID(), parent)
	if err != nil {
	return nil, err
	}

	v1Img.ID = v1ID.Encoded()
	if parent != "" {
	v1Img.Parent = parent.Encoded()
	}

	LDFLAGS="${LDFLAGS} \
	-X \"github.com/docker/docker/dockerversion.Version=${VERSION}\" \
	-X \"github.com/docker/docker/dockerversion.GitCommit=${GITCOMMIT}\" \
	-X \"github.com/docker/docker/dockerversion.BuildTime=${BUILDTIME}\" \
	-X \"github.com/docker/docker/dockerversion.PlatformName=${PLATFORM}\" \
	-X \"github.com/docker/docker/dockerversion.ProductName=${PRODUCT}\" \
	-X \"github.com/docker/docker/dockerversion.DefaultProductLicense=${DEFAULT_PRODUCT_LICENSE}\" "

	$ldflags = "-X 'github.com/docker/docker/dockerversion.Version="+$dockerVersion+"'"
	$ldflags += " -X 'github.com/docker/docker/dockerversion.GitCommit="+$gitCommit+"'"
	$ldflags += " -X 'github.com/docker/docker/dockerversion.BuildTime="+$env:BUILDTIME+"'"
	$ldflags += " -X 'github.com/docker/docker/dockerversion.PlatformName="+$env:PLATFORM+"'"
	$ldflags += " -X 'github.com/docker/docker/dockerversion.ProductName="+$env:PRODUCT+"'"
	$ldflags += " -X 'github.com/docker/docker/dockerversion.DefaultProductLicense="+$env:DEFAULT_PRODUCT_LICENSE+"'"

	tsh := &tarsumHash{hdr: hdr, Hash: sha256.New()}
	tsh.Reset() // initialize header
	return tsh, nil
	}

	type tarsumHash struct {
	hash.Hash
	hdr *tar.Header
	}

	// Reset resets the Hash to its initial state.
	func (tsh *tarsumHash) Reset() {
	// comply with hash.Hash and reset to the state hash had before any writes
	tsh.Hash.Reset()
	tarsum.WriteV1Header(tsh.hdr, tsh.Hash)
	}

		"github.com/docker/docker/internal/unshare"
		"github.com/docker/docker/daemon/internal/unshare"

Move remaining image packages to daemon #50446

Move remaining image packages to daemon #50446

Conversation

dmcgowan commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

thaJeztah Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

thaJeztah Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thaJeztah Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

dmcgowan Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

thaJeztah Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

vvoland left a comment

Choose a reason for hiding this comment

Uh oh!

austinvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

dmcgowan commented Jul 25, 2025

Uh oh!

austinvazquez commented Jul 25, 2025

Uh oh!

dmcgowan commented Jul 25, 2025

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jul 25, 2025

Uh oh!

thaJeztah commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

dmcgowan commented Jul 17, 2025 •

edited

Loading