Support RSA SHA-2 (RFC8332) signatures for SSH authentication #916
Description
System Information
Linux distribution
This affects systems were the remote server is running Fedora 33, 34, 35, CentOS Stream 9 and RHEL 8 with FIPS.
Terraform and provider versions
Terraform v1.1.2
on darwin_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.12Description of Issue/Question
Attempting to connect to any of the affected OpenSSH configurations — in my case, default Fedora 34 ssh config — RSA based ssh authentication will fail with the error failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain and the servers sshd.log will report the following:
Dec 28 11:37:12 hylfing.westmore sshd[843630]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Dec 28 11:37:12 hylfing.westmore sshd[843630]: Connection closed by authenticating user davidalger 172.16.0.138 port 54755 [preauth]Fedora 33 updated the system-wide crypto policy to disallow SHA-1 hashes in signatures and OpenSSH 8.8 (released on 2021-09-26) disables the ssh-rsa signature scheme by default as well (http://www.openssh.com/txt/release-8.7) so the change will eventually trickle into other Linux families as well if it hasn't already. In particular, "ssh-rsa" keys are capable of signing using "rsa-sha2-256" (RSA/SHA256), "rsa-sha2-512" (RSA/SHA512) and "ssh-rsa" (RSA/SHA1). Only the last of these is being turned off by default.
This explains why it was reported that ed25519 keys worked with ssh-agent support added in 0.6.11 while RSA keys continued to fail (#864 (comment)) as the commenter indicated (in previous comment) he was testing with a Fedora 34 server, one which would have ssh-rsa disabled by default.
Setup
terraform {
required_version = "~> 1.1"
required_providers {
libvirt = {
source = "dmacvicar/libvirt"
version = "~> 0.6"
}
}
}
provider "libvirt" {
uri = "qemu+ssh://davidalger@hylfing/system?sshauth=agent&socket=/var/run/libvirt/libvirt-sock"
}
resource "libvirt_volume" "fcos35" {
name = "fcos35"
source = "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20211203.3.0/x86_64/fedora-coreos-35.20211203.3.0-qemu.x86_64.qcow2.xz"
}Steps to Reproduce Issue
Terraform debug output:
2021-12-29T10:38:30.543-0600 [INFO] Terraform version: 1.1.2
2021-12-29T10:38:30.543-0600 [INFO] Go runtime version: go1.17.2
2021-12-29T10:38:30.543-0600 [INFO] CLI args: []string{"/usr/local/Cellar/tfenv/2.2.2/versions/1.1.2/terraform", "plan"}
2021-12-29T10:38:30.543-0600 [DEBUG] Attempting to open CLI config file: /Users/davidalger/.terraformrc
2021-12-29T10:38:30.544-0600 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-12-29T10:38:30.544-0600 [DEBUG] checking for credentials in "/Users/davidalger/.terraform.d/plugins"
2021-12-29T10:38:30.544-0600 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-12-29T10:38:30.544-0600 [DEBUG] will search for provider plugins in /Users/davidalger/.terraform.d/plugins
2021-12-29T10:38:30.544-0600 [DEBUG] ignoring non-existing provider search directory /Users/davidalger/Library/Application Support/io.terraform/plugins
2021-12-29T10:38:30.544-0600 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021-12-29T10:38:30.544-0600 [INFO] CLI command args: []string{"plan"}
2021-12-29T10:38:30.545-0600 [DEBUG] New state was assigned lineage "81062f15-d37c-b4e5-9bf2-8aec4bfcc378"
2021-12-29T10:38:30.607-0600 [DEBUG] checking for provisioner in "."
2021-12-29T10:38:30.607-0600 [DEBUG] checking for provisioner in "/usr/local/Cellar/tfenv/2.2.2/versions/1.1.2"
2021-12-29T10:38:30.607-0600 [DEBUG] checking for provisioner in "/Users/davidalger/.terraform.d/plugins"
2021-12-29T10:38:30.609-0600 [INFO] backend/local: starting Plan operation
2021-12-29T10:38:30.611-0600 [DEBUG] created provider logger: level=debug
2021-12-29T10:38:30.611-0600 [INFO] provider: configuring client automatic mTLS
2021-12-29T10:38:30.619-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12]
2021-12-29T10:38:30.621-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56983
2021-12-29T10:38:30.622-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12
2021-12-29T10:38:30.636-0600 [INFO] provider.terraform-provider-libvirt_v0.6.12: configuring server automatic mTLS: timestamp=2021-12-29T10:38:30.636-0600
2021-12-29T10:38:30.676-0600 [DEBUG] provider: using plugin: version=5
2021-12-29T10:38:30.676-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: plugin address: address=/var/folders/b5/88xzlq3n61jdf_f30wlkkg580000gn/T/plugin073562578 network=unix timestamp=2021-12-29T10:38:30.676-0600
2021-12-29T10:38:30.706-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2021-12-29T10:38:30.706-0600 [DEBUG] No provider meta schema returned
2021-12-29T10:38:30.709-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56983
2021-12-29T10:38:30.709-0600 [DEBUG] provider: plugin exited
2021-12-29T10:38:30.709-0600 [DEBUG] Building and walking validate graph
2021-12-29T10:38:30.709-0600 [DEBUG] ProviderTransformer: "libvirt_volume.fcos35" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2021-12-29T10:38:30.709-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.fcos35" references: []
2021-12-29T10:38:30.709-0600 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2021-12-29T10:38:30.709-0600 [DEBUG] Starting graph walk: walkValidate
2021-12-29T10:38:30.710-0600 [DEBUG] created provider logger: level=debug
2021-12-29T10:38:30.710-0600 [INFO] provider: configuring client automatic mTLS
2021-12-29T10:38:30.719-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12]
2021-12-29T10:38:30.722-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56984
2021-12-29T10:38:30.723-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12
2021-12-29T10:38:30.736-0600 [INFO] provider.terraform-provider-libvirt_v0.6.12: configuring server automatic mTLS: timestamp=2021-12-29T10:38:30.736-0600
2021-12-29T10:38:30.772-0600 [DEBUG] provider: using plugin: version=5
2021-12-29T10:38:30.772-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: plugin address: network=unix address=/var/folders/b5/88xzlq3n61jdf_f30wlkkg580000gn/T/plugin216229823 timestamp=2021-12-29T10:38:30.772-0600
2021-12-29T10:38:30.802-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2021-12-29T10:38:30.802-0600 [DEBUG] No provider meta schema returned
2021-12-29T10:38:30.806-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56984
2021-12-29T10:38:30.806-0600 [DEBUG] provider: plugin exited
2021-12-29T10:38:30.806-0600 [INFO] backend/local: plan calling Plan
2021-12-29T10:38:30.807-0600 [DEBUG] Building and walking plan graph for NormalMode
2021-12-29T10:38:30.807-0600 [DEBUG] ProviderTransformer: "libvirt_volume.fcos35 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2021-12-29T10:38:30.808-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.fcos35 (expand)" references: []
2021-12-29T10:38:30.808-0600 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2021-12-29T10:38:30.808-0600 [DEBUG] Starting graph walk: walkPlan
2021-12-29T10:38:30.809-0600 [DEBUG] created provider logger: level=debug
2021-12-29T10:38:30.809-0600 [INFO] provider: configuring client automatic mTLS
2021-12-29T10:38:30.818-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12]
2021-12-29T10:38:30.820-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56985
2021-12-29T10:38:30.820-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12
2021-12-29T10:38:30.834-0600 [INFO] provider.terraform-provider-libvirt_v0.6.12: configuring server automatic mTLS: timestamp=2021-12-29T10:38:30.834-0600
2021-12-29T10:38:30.870-0600 [DEBUG] provider: using plugin: version=5
2021-12-29T10:38:30.870-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: plugin address: address=/var/folders/b5/88xzlq3n61jdf_f30wlkkg580000gn/T/plugin702862148 network=unix timestamp=2021-12-29T10:38:30.870-0600
2021-12-29T10:38:30.894-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2021-12-29T10:38:30.895-0600 [DEBUG] No provider meta schema returned
2021-12-29T10:38:30.896-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: 2021/12/29 10:38:30 [DEBUG] Configuring provider for 'qemu+ssh://davidalger@hylfing/system?sshauth=agent&socket=/var/run/libvirt/libvirt-sock': &{map[uri:0xc0000b63c0] <nil> <nil> 0xc0005a6ba0 map[] <nil> 0xc0005a6c20 0xc00019bef0 0xc000150280 false map[] {1 {0 0}} false false}
2021-12-29T10:38:31.162-0600 [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-12-29T10:38:31.163-0600 [INFO] backend/local: plan operation completed
╷
│ Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
│
│ with provider["registry.terraform.io/dmacvicar/libvirt"],
│ on main.tf line 12, in provider "libvirt":
│ 12: provider "libvirt" {
│
╵
2021-12-29T10:38:31.166-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56985
2021-12-29T10:38:31.166-0600 [DEBUG] provider: plugin exitedsshd.log from the server showing why the handshake failed
Dec 28 11:37:12 hylfing.westmore sshd[843630]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Dec 28 11:37:12 hylfing.westmore sshd[843630]: Connection closed by authenticating user davidalger 172.16.0.138 port 54755 [preauth]Additional information:
root@hylfing:~# ssh -Q PubkeyAcceptedAlgorithms | grep rsa
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512-cert-v01@openssh.com
root@hylfing:~# sshd -T | grep -i PubkeyAcceptedAlgorithms
pubkeyacceptedalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
This may be the root cause for some of the issues noted in #886
Possible solution:
Crypto packages should now support RSA SHA-2 (RFC8332) signatures, so upgrading may be all that is needed to support them. Please see golang/go#37278.
For now I'm going to workaround the issue by adding an ed25519 key to my agent, allowing the plan to succeed:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# libvirt_volume.fcos35 will be created
+ resource "libvirt_volume" "fcos35" {
+ format = (known after apply)
+ id = (known after apply)
+ name = "fcos35"
+ pool = "default"
+ size = (known after apply)
+ source = "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20211203.3.0/x86_64/fedora-coreos-35.20211203.3.0-qemu.x86_64.qcow2.xz"
}
Plan: 1 to add, 0 to change, 0 to destroy.