Failing authentication for jj git fetch ... / jj git clone ... / etc. with a FIDO2 (resident) key #4591
Description
Description
I know, I might have encountered a very special and specific scenario.
Nevertheless, I can't get any git remote command to work properly.
For SSH authentication, I'm using a Yubikey using a FIDO2 resident key.
In particular, the key was generated using the following command ssh-keygen -t ed25519-sk -O resident -O user=git -O application=ssh:git -O verify-required.
As I read on several occasions, the verify-required option can sometimes cause problems (it not only requires touch but also the entry of a PIN), so I also tried without: ssh-keygen -t ed25519-sk -O resident -O user=git -O application=ssh:git.
After that also didn't work, I tried a non-resident key with ssh-keygen -t ed25519-sk -O user=git -O application=ssh:git, still to no avail.
In every scenario, the key was present at ~/.ssh/id_ed25519_sk and present in the ssh-agent confirmed using ssh-add -L.
Each time I get prompted for touching the hardware token (or in case of verify-required also for the PIN) without issue but then it fails with
Error: failed to authenticate SSH session: ; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?
Of course I also tried the mentioned command: ssh -F /dev/null git@github.com -vv without any issues.
It should also go without saying that doing things like git fetch or git pull work without any issues.
> ssh -F /dev/null git@github.com -vv
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
debug1: Reading configuration data /dev/null
debug2: resolving "github.com" port 22
debug1: Connecting to github.com [140.82.121.3] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\info/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\info/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa_sk type -1
debug1: identity file C:\\Users\\info/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\info/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_ed25519_sk type 12
debug1: identity file C:\\Users\\info/.ssh/id_ed25519_sk-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\info/.ssh/id_xmss-cert type -1
debug1: identity file C:\\Users\\info/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\info/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_9.5
debug1: Remote protocol version 2.0, remote software version babeld-3d60e350f
debug1: compat_banner: no match: babeld-3d60e350f
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to github.com:22 as 'git'
debug1: load_hostkeys: fopen C:\\Users\\info/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,kex-strict-s-v00@openssh.com
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
debug1: load_hostkeys: fopen C:\\Users\\info/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'github.com' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\info/.ssh/known_hosts:1
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug2: get_agent_identities: ssh_agent_bind_hostkey: invalid format
debug1: get_agent_identities: agent returned 1 keys
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ed25519_sk ED25519-SK SHA256:EiWoeWPBZv3JXnLqAJd+Y+1iHkutP7VXdmXzoj5EM9w authenticator agent
debug1: Will attempt key: C:\\Users\\info/.ssh/id_rsa
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ecdsa_sk
debug1: Will attempt key: C:\\Users\\info/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\info/.ssh/id_xmss
debug1: Will attempt key: C:\\Users\\info/.ssh/id_dsa
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\info/.ssh/id_ed25519_sk ED25519-SK SHA256:EiWoeWPBZv3JXnLqAJd+Y+1iHkutP7VXdmXzoj5EM9w authenticator agent
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: C:\\Users\\info/.ssh/id_ed25519_sk ED25519-SK SHA256:EiWoeWPBZv3JXnLqAJd+Y+1iHkutP7VXdmXzoj5EM9w authenticator agent
Authenticated to github.com ([140.82.121.3]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug2: channel 0: send open
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: ENABLE_VIRTUAL_TERMINAL_INPUT is supported. Reading the VTSequence from console
debug1: ENABLE_VIRTUAL_TERMINAL_PROCESSING is supported. Console supports the ansi parsing
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching C:\\Users\\info/.ssh/known_hosts for github.com / (none)
debug1: client_input_hostkeys: searching C:\\Users\\info/.ssh/known_hosts2 for github.com / (none)
debug1: client_input_hostkeys: hostkeys file C:\\Users\\info/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug2: channel 0: request shell confirm 1
debug1: pledge: fork
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 32000 rmax 35000
debug2: channel_input_status_confirm: type 100 id 0
PTY allocation request failed on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug2: channel 0: rcvd ext data 3
debug2: channel 0: rcvd ext data 9
debug2: channel 0: rcvd ext data 2
Hi DustVoice! debug2: channel 0: written 2044404432910 to efd 6
debug2: channel 0: rcvd ext data 76
debug2: channel 0: rcvd ext data 1
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: rcvd close
debug2: chan_shutdown_read: channel 0: (i0 o1 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug1: channel 0: free: client-session, nchannels 1
Connection to github.com closed.
Transferred: sent 2192, received 2668 bytes, in 6.0 seconds
Bytes per second: sent 368.3, received 448.3Executing jj with the --debug flag produces the following output
> jj git fetch --debug
2024-10-06T15:36:21.468008Z INFO jj_cli::cli_util: debug logging enabled
2024-10-06T15:36:21.510254Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.download
2024-10-06T15:36:22.242154Z INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key_from_agent username="git"
2024-10-06T15:36:43.313220Z INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}:get_ssh_keys{_username="git"}: jj_cli::git_util: found ssh key path="C:\\Users\\info\\.ssh\\id_ed25519_sk"
2024-10-06T15:36:43.313429Z INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_bookmark: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key username="git" path="C:\\Users\\info\\.ssh\\id_ed25519_sk"
Error: failed to authenticate SSH session: ; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?As everything works with git and even non-resident keys don't work with jj I'm assuming that the ed25519-sk format might somehow be the culprit?
It's weird as in my experience, a non-support of this SSH feature normally results in nothing working at all, but in this case, with jj the PIN entry opens, and I'm able to touch the key.
I also researched in the libssh2 repo, but every related issue I could find regarding FIDO2 and the key format has apparently been resolved and closed.
It is probably also worth noting that signing with this kind of SSH key is apparently working fine, as jj git init --colocate prompted me for my PIN/touch a couple of times and then successfully completed.
[signing]
sign-all = true
backend = "ssh"
key = "~/.ssh/id_ed25519_sk.pub"
[signing.ssh.backends.ssh]
allowed-signers = "~/.ssh/allowed_signers"Thank you for your awesome work, I hope I will soon come to enjoy this remarkable piece of work.
Steps to Reproduce the Problem
- Generate a
ed25519-skkey with a hardware security key usingssh-keygen - Put the file into your
~/.ssh/directory with the default name (id_ed25519_sk/id_ed25519_sk.pub) - Add it to your
ssh-agentusingssh-add(ssh-add ~/.ssh/id_ed25519_sk) - Add the public key
id_ed25519_sk.pubto your GitHub account as an Authentication key - Invoke
jj git fetch - Confirm user presence on the key (often times touching and/or a PIN entry)
- Watch
jjfail
Expected Behavior
jj should successfully authenticate against the git server (GitHub in this instance) and be able to push, pull and clone without issue.
Actual Behavior
jj fails without any hints of why it fails after the user already confirmed their presence.
Locating the key in the ssh-agent, as well as the key file itself, seemed to be successful.
Specifications
- Platform: Windows 11 Pro (Version 10.0.22631 Build 22631), YubiKey 5 Nano (firmware version 5.4.3)
- Scoop versions: git-without-openssh (version 2.46.2.windows.1), openssh (version 9.5.0.0p1); (
gitandopensshare installed seperately as thesshversion bundled withgiton Windows is too outdated to properly support FIDO2) - Version:
jjversion 0.22.0, both fromscoopandcargowith same result