Skip to content

update 3rd party libraries, part2 #3707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Apr 2, 2025
Merged

update 3rd party libraries, part2 #3707

merged 17 commits into from
Apr 2, 2025

Conversation

georgeliao
Copy link
Contributor

@georgeliao georgeliao commented Sep 30, 2024

close #3568
MULTI-1176

git submodule
1. vcpkg, updated to 2025.01.13.

vcpkg managed:
1. grpc, 1.52.1-> 1.68.2.

The vcpkg grpc build recipe (located in <multipass_path>/3rd-party/vcpkg-ports/grpc) particularly portfile.cmake, is based on the standard grpc build recipe found in the directory <multipass_path>/3rd-party/vcpkg/ports/grpc. Therefore, an effective way to view the actual change is by comparing these two directories using tools like Beyond Compare or diff.

Most changes are centered around one thing, that is eliminating unneeded library build. As a result, it significantly reduced the grpc build time. To achieve this, remove_unneeded_lib_custom.patch was introduced. To review the content of the patch, you can compare it with the original patch applied to grpc 1.52.

Although maintaining this patch can be cumbersome, the build time savings justify the effort. The grpc build time was reduced from 9 mins 53 seconds to 4 mins 40 seconds on my 20 hardware threads machine.

Another deviation from the standard build recipe is the removal of openssl from the 3rd-party/vcpkg-ports/grpc/vcpkg.json, this follows the setup of the old version. The purpose is to prevent vcpkg pulls openssl. Consequently, the cmake will find system installed openssl. As to whether we want to migrate to the vcpkg openssl from the system openssl, that can be discussed within the team and done in another PR.

Additionally, there are some api calls changes on multipass side. Those are made to accomodate some grpc' code changes in this duration which broke the backward compatibility. The particular commit is this one. There is also another github issue which has relevant discussions about this. This change affects grpc version 1.57.0 and onwards. The fix we adopt is the one mentioned in the github issue, where they call grpc.insecure_channel(srv_endpoint, options=(('grpc.default_authority', 'localhost'),)) on python client to overwrite the default authority, the C++ equivalent to this would be changing grpc::CreateChannel calls to grpc::CreateCustomChannel calls with channel arguments, see below code snippet for details

    grpc::ChannelArguments channel_args;
    channel_args.SetString(GRPC_ARG_DEFAULT_AUTHORITY, "localhost");

    // Use CreateCustomChannel to apply ChannelArguments
    auto channel = grpc::CreateCustomChannel(
        server_address, grpc::InsecureChannelCredentials(), channel_args);

Be aware that this change affects all types of address resolvers, that probably is fine in the use cases of multipass.

snap:

1. sshfs, will be done it in a different PR. 

One small note: the update of vcpkg changes the file structure in scripts folder, so you will need to re-run the bootstrap script (./bootstrap-vcpkg.sh on linux and ./bootstrap-vcpkg.bat on windows) in <multipass_src_path>/3rd-party/vcpkg directory.

@georgeliao georgeliao marked this pull request as draft September 30, 2024 16:23
@georgeliao georgeliao force-pushed the update_3rd_lib_version branch from ca31a3a to 25a88f3 Compare October 2, 2024 10:49
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch from f7603a6 to 6481118 Compare October 2, 2024 13:39
@georgeliao georgeliao force-pushed the update_3rd_lib_version branch from b254d90 to 2a93369 Compare October 2, 2024 15:53
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch from 6481118 to f39584d Compare October 3, 2024 08:45
@georgeliao georgeliao force-pushed the update_3rd_lib_version branch from 2a93369 to d79d2c9 Compare October 3, 2024 09:36
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch 2 times, most recently from a45865d to 5bb9771 Compare October 3, 2024 09:50
Base automatically changed from update_3rd_lib_version to main October 3, 2024 13:50
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch 2 times, most recently from 84e7715 to 1766a7b Compare October 4, 2024 08:46
Copy link

codecov bot commented Oct 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.34%. Comparing base (5385fac) to head (e795fce).
Report is 3408 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3707   +/-   ##
=======================================
  Coverage   89.33%   89.34%           
=======================================
  Files         260      260           
  Lines       14735    14738    +3     
=======================================
+ Hits        13164    13167    +3     
  Misses       1571     1571           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ricab ricab added this to the 1.15.0 milestone Oct 7, 2024
vcpkg.json Outdated
@@ -1,5 +1,5 @@
{
"builtin-baseline": "ca7b1b15f548c25c766360593a2c732d56ed0133",
"builtin-baseline": "c82f74667287d3dc386bce81e44964370c91a289",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

builtin-baseline field defines the versions of the dependencies of our vcpkg packages. In this particular case, grpc needs this update to get his dependency versions right because of his own update. The value of builtin-baseline field is simply the commit hash of the vcpkg repo.

@georgeliao
Copy link
Contributor Author

georgeliao commented Nov 6, 2024

Hi @Saviq @townsend2010

Sorry to bother you guys. Recently, multipass team has been dealing with rebasing the patches (code commits on the top of the release tag) of grpc after updating grpc to newer version. It has been a painful process due to the size of the patches and conflicts with the grpc refactors and changes. This really got us thinking what is the exact reason to keep these patches. Why multipass has a unique use case from the regular ones?

By looking at commit messages and the code, my superficial understanding is that. Multipass wants to support skipping verifying server certificate. Because of that, the grpc_ssl_server_certificate_request_type enum is added and all the corresponding changes are made. But why exactly do we need that is still not clear to me, meaning why does multipass uniquely need this skip the server certificate verification option.

Any information would be appreciated in case you guys know about it, thanks a lot.

@Saviq
Copy link
Collaborator

Saviq commented Nov 6, 2024

why does multipass uniquely need this skip the server certificate verification option

Basically to support SSH-style authorization. Both sides generate their encryption keys in isolation, without a CA between them. All communication is encrypted, and the key fingerprints are what determines whether they want to talk to one other on a higher level, not whether the certificates are valid or not.

@georgeliao
Copy link
Contributor Author

georgeliao commented Nov 12, 2024

Basically to support SSH-style authorization. Both sides generate their encryption keys in isolation, without a CA between them. All communication is encrypted, and the key fingerprints are what determines whether they want to talk to one other on a higher level, not whether the certificates are valid or not.

Thanks for the reply. But why exactly does multipass have to use the SSH-style authorization? The standard TLS/mTLS always requires the server certificate and verifies it. As a result, grpc api only provides GRPC_SSL_REQUEST_SERVER_CERTIFICATE_AND_VERIFY but not GRPC_SSL_REQUEST_SERVER_CERTIFICATE_BUT_DONT_VERIFY and it seems to make sense. The orthodox way based on the documentation of doing TLS/mTLS grpc is to fill in the pem_root_certs field of grpc::SslCredentialsOptions object on the client side, and later it will be used to verify the server cert and key pair internally.

My wondering here is whether this orthodox way has some inadequacies we are not aware of or it somehow does not fit in the multipass use case. Any info will be appreciated, thanks.

@georgeliao georgeliao marked this pull request as ready for review November 18, 2024 11:34
@georgeliao georgeliao marked this pull request as draft November 18, 2024 11:35
@townsend2010
Copy link
Contributor

My wondering here is whether this orthodox way has some inadequacies we are not aware of or it somehow does not fit in the multipass use case. Any info will be appreciated, thanks.

Hey @georgeliao,

As @Saviq alluded to, it was done this way to avoid having to use a CA and generate valid SSL certificates. In the early days when this was implemented, we found it to be easier to just modify gRPC and carry the patches than to deal with a CA. If you all want to deal with a CA and generate valid SSL certs in order to use what gRPC already supports, then that's a technical decision the Multipass team will need to make. Maybe it's easier now than in 2018. 🤷‍♂️ Care will also need to be taken with the whole authorization plumbing, but I think that should still work.

@Saviq
Copy link
Collaborator

Saviq commented Nov 18, 2024

This was the only way to have clients other than the first/default one to get access to the daemon simply by providing a shared passphrase. If you go for the CA approach, you have to generate a key pair and a certificate signing request and ask someone with access to the CA (admin) to sign your CSR and provide you back with the signed certificate. There's also no path to accepting the first client like we do today on Linux and macOS, you'd need to somehow get through the above dance in the installation phase.

Using a passphrase is just a more user-friendly way to achieving the same level of security, is all (one missing bit in the current scheme is managing a list of accepted daemon certificates, .ssh/known_hosts-style).

@georgeliao
Copy link
Contributor Author

@townsend2010
Thanks a lot for the information. Now we understand the context much better.

@ricab ricab modified the milestones: 1.15.0, 1.15.1 Nov 20, 2024
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch from 816e265 to 114fe3f Compare December 4, 2024 09:31
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch from 114fe3f to 6b9d382 Compare January 8, 2025 15:59
@ricab ricab dismissed stale reviews from Sploder12 and xmkg March 31, 2025 09:52

The base branch was changed.

georgeliao and others added 17 commits March 31, 2025 12:12
…, which is non-intrusiveserver address resolver from intrusive change to
Co-authored-by: Ricardo Abreu <6698114+ricab@users.noreply.github.com>
Signed-off-by: George Liao <georgeliaojia@gmail.com>
@georgeliao georgeliao force-pushed the update_3rd_lib_part2 branch from fc97bbb to e795fce Compare March 31, 2025 11:02
@georgeliao
Copy link
Contributor Author

@ricab , @xmkg @Sploder12 can any of you approve this again so we can start the merge queue?

@georgeliao
Copy link
Contributor Author

@Sploder12 Can you do some sanity testing on various platforms since this PR has been rebased several times before the approval? Thanks.

Copy link
Contributor

@Sploder12 Sploder12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Windows with installer, Snap and not-Snap Ubuntu gave me no issues while testing!

@georgeliao
Copy link
Contributor Author

I have tested on macos with the installer. Things are fine.

@georgeliao georgeliao added this pull request to the merge queue Apr 2, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 2, 2025
@georgeliao georgeliao added this pull request to the merge queue Apr 2, 2025
Merged via the queue into main with commit f825dee Apr 2, 2025
16 checks passed
@georgeliao georgeliao deleted the update_3rd_lib_part2 branch April 2, 2025 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update all our dependencies
6 participants