Skip to content

[CI]: Discussion: should we drop testing inside a container? #3858

@apostasie

Description

@apostasie

Our testing rig is designed around a container, and the CI itself is currently testing nerdctl purely inside a docker container.

This may be a good idea for server-side components that are also deployed and used inside containers.
It also may seem like a good idea to more easily "reproduce" specific target environments.

However, nerdctl is not a server component used in controlled environments. It's a cli, most of the time ran directly on (a variety of) hosts that we do not control.

It is very obvious that nobody is running the test suite directly on the host.
Because it fails. Despite recent efforts to fix it, it is still not passing on the most obvious and simplest environment of all: the default lima install.

Furthermore, by limiting developers' testing to that specific, artificially controlled container environment, we get a false sense of success, and tend to think that things are working, while really they are not.

I am suggesting that we stop encouraging testing inside the docker container, and instead that we re-design our testing to just run natively, and that we encourage contributors and users alike to test in their specific environment.

I am also suggesting that we remove the extra level of indirection on the CI if feasible.

Thoughts?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions