The libssh2_userauth_publickey_frommemory() function accepts private key data in the form of an arbitrary buffer and its length. From the documentation for libssh2_userauth_publickey_frommemory():

privatekeydata - Buffer containing the contents of a private key file.
privatekeydata_len - Length of private key data.

... but that buffer is passed down the chain via the filename member of a struct privkey_file:

When that buffer is ultimately processed by sign_frommemory(), the originally-provided buffer length has been effectively discarded, and its length is instead calculated as if it were a string using strlen():

Depending on the size of the block and memory garbage, this will overread the buffer and give nondeterministic results.