Hey everyone!

We would like to raise our concerns about the upcoming removal of the GRPC_ENFORCE_ALPN_ENABLED environment variable in version 1.68.0 of the gRPC library, which is currently slated to enforce the proper usage of ALPN permanently (without an opt-out).

In versions prior to 1.67.0, this feature was controlled via the GRPC_ENFORCE_ALPN_ENABLED environment variable, defaulting to false. In 1.67.0, the default was changed to true, and in 1.68.0, the plan is to remove the variable entirely and make the behavior mandatory.

We understand and support the need for proper configurations, but this change poses significant challenges for our usage. We develop Talos OS, where our clients upgrade their systems gradually, independently from us, not all at once, sometimes skipping several versions at one moment. If this change goes through as planned, it would prevent newer nodes from communicating with older nodes, potentially causing upgrade failures (such as workers not able to communicate with control plane nodes).

Additionally, we also develop another product, Omni, which clients can use in a self-hosted mode. In scenarios where older versions of Omni are running with gRPC <= 1.67.x (where GRPC_ENFORCE_ALPN_ENABLED is not set), newer versions of Talos OS using gRPC >= 1.68.0 will lose the ability to manage these clusters of those nodes.

This situation could create significant disruptions for our clients. We believe that we are not alone in facing this challenge, as many microservice-based architectures would similarly struggle to update all services simultaneously, leading to potential downtime and service interruptions.

We kindly request that you reconsider the removal of the GRPC_ENFORCE_ALPN_ENABLED variable in version 1.68.y beyond. It would also be great if you also provide a way to enforce old behavior using the Go code (calling a function in init perhaps?), rather than environment variables.

This change is very important to us and to others in the community who rely on gradual, seamless upgrades.