I'm getting a -44 error code and message of "Unable to ask for ssh-userauth service" when using wolfssl as the encryption engine during ssh handshake. I've tracked the error to this code on line 584 of openssl.c:

if(IS_LAST(firstlast)) {
  /* This is the last block.
     encrypt: compute tag, if applicable
     decrypt: verify tag, if applicable
     in!=NULL is equivalent to EVP_CipherUpdate
     in==NULL is equivalent to EVP_CipherFinal */
#ifdef HAVE_OPAQUE_STRUCTS
  ret = EVP_Cipher(*ctx, NULL, NULL, 0); /* final */
#else
  ret = EVP_Cipher(ctx, NULL, NULL, 0); /* final */
#endif

The comment indicates the expectation that EVP_CipherFinal should be equivalent to EVP_Cipher when finalizing the encryption. The EVP_Cipher method resolves to wolfSSL_EVP_Cipher on line 7080 of the wolfssl/wolfcrypt/src/evp.c file. There a check of NULL on the incoming buffers returns a WOLFSSL_FATAL_ERROR. After modifying the code to follow the comment, EVP_CipherFinal is used instead of EVP_Cipher, then the encryption seems to work as expected:

int final_length = 0;
ret = EVP_CipherFinal(*ctx, buf + aadlen, &final_length); /* final */

I do not know enough about these internals to know if this is the correct solution or not and would appreciate any assistance in getting this fixed. The comment suggests a break in the interface expectations with wolfssl.

To Reproduce
Extract libssh2 1.11.0, wolfssl 5.6.4. Build wolfssl. Build libssh2 to use wolfssl. Attempt to connect to an ssh server. In my case OpenSSH 8.9.

Expected behavior
For the encryption to work and the ssh handshake to complete.

Version

• OS and version: Linux Ubuntu 22.04.3 LTS
• libssh2 version: 1.11.0
• crypto backend and version: wolfSSL 5.6.4

Additional context
Add any other context about the problem here.