I suspect that operations that implement the kubernetes_manifest is attempting to connect to non-routable IP from my local machine by not using the port-forwarded https proxy that I defined with KUBE_PROXY_URL. Creating a kubernetes_namespace with this setup works fine. My module with kubernetes_manifest resources works fine in a development cluster that has publicly accessible nodes and API endpoint without a KUBE_PROXY_URL defined.

I guess the workaround I'll have to try is converting the manifests to HCL.

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.2.2
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.22.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.11.0

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

Relevant configuration section:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.22.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.11.0"
    }
  }
  backend "gcs" {
    bucket = "MY_BUCKET"
    prefix = "MY_CLUSTER/CLUSTER_SERVICES"
  }
}

data "google_client_config" "provider" {}

provider "kubernetes" {
  host                   = var.kubernetes_host
  token                  = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(var.kubernetes_host_ca_certificate)
}

Steps to Reproduce

1. Create a Kubernetes cluster that is not accessible from the Internet like a Google Cloud cluster with private nodes and private endpoint.
2. Create a bastion host in a subnet that is authorized to access the Kubernetes endpoint. Bastion runs a http proxy like tinyproxy.
3. Open up a SSH connection to the bastion with port forwarding to the private endpoint (e.g., -L 8888:localhost:8888)
4. Run Terraform like KUBE_PROXY_URL="http://localhost:8888" terraform ....

Expected Behavior

Terraform should be able to create all kinds of kubernetes resources.

Actual Behavior

I can create a kubernetes_namespace, but when setting up third-party services like cert-manager through kubernetes_manifest resources that created with yamldecode(file(...)), I get errors like the following:

2022-06-03T16:08:49.216-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.11.0_x5: 2022-06-03T16:08:49.157-0500 [DEBUG] [InvalidClientConfiguration]: Config="&rest.Config{Host:\"https://[[PRIVATE_K8S_IP]]\", APIPath:\"\", ContentConfig:rest.ContentConfig{AcceptContentTypes:\"\", ContentType:\"\", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:(*serializer.negotiatedSerializerWrapper)(0xc001bfb5e0)}, Username:\"\", Password:\"\", BearerToken:\"--- REDACTED ---\", BearerTokenFile:\"\", Impersonate:rest.ImpersonationConfig{UserName:\"\", UID:\"\", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:\"\", CertFile:\"\", KeyFile:\"\", CAFile:\"\", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8{[[REDACTED]]}, NextProtos:[]string(nil)}, UserAgent:\"\", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}"
2022-06-03T16:08:49.217-0500 [ERROR] vertex "module.cluster-services.module.cert_manager.module.stage_1.kubernetes_manifest.file[\"manifests/ClusterRoleBinding-cert-manager-cainjector.yml\"]" error: Invalid configuration for API client

References

• similar to proxy_url value of https://localhost:<port> throws Unable to connect to the server: proxyconnect tcp: tls: first record does not look like a TLS handshake #1653, but it is clear that using the proxy works since creating and refreshing kubernetes_namespace resources works fine
• bastion discussion in Support for running port-forward while Terraform is operating #812

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment