grpc: Add AWS IAM grpc credentials extension #5546

lavignes · 2019-01-08T23:07:44Z

grpc: Add AWS IAM grpc credentials extension

Description: As part of leveraging Envoy in AWS App Mesh, we implemented a gRPC credentials extension that uses AWS IAM auth. Our original implementation added a dependency on the AWS SDK and libcurl to Envoy. This implementation leverages components within Envoy to fetch credentials from various sources and sign xDS requests for communicating with our App Mesh envoy management server. There are a few more details in #5215.
The request signer and credential providers themselves are reusable components that could be used to implement other extensions that need to authenticate with AWS services.
Risk Level: Medium
Testing: Added unit tests, + manual testing. Planning to add an integration test on the extension.
Docs Changes: Added in new protos.
Release Notes: Added line about added extension.

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

htuch · 2019-01-09T04:27:57Z

@lavignes thanks for this contribution. I have two high level structural comments before diving into the detail:

Does source/common/aws belong in extensions instead, given this is a gRPC credential extension?
The PR is a great WiP and reference, but for review I'd like this to be broken down into separate PRs. Small PRs are great for all the usual reasons, we'll be able to merge your code faster, review faster and be happier as maintainers if we can do that. Can you propose a patch series to make this happen?

htuch · 2019-01-09T04:31:40Z

Also, quick aside; we're about to gain libcurl as a transitive dependency necessary for OpenCensus, see #5387.

ggreenway · 2019-01-09T21:08:03Z

Does source/common/aws belong in extensions instead, given this is a gRPC credential extension?

+1 to this. This should definitely live in extentions, with appropriate interfaces for common code to use it.

lavignes · 2019-01-10T21:26:26Z

@htuch

Does source/common/aws belong in extensions instead, given this is a gRPC credential extension?

This is a result of the earlier version of this code. We had to make changes to Main to initialize the AWS SDK. Easy enough to move to source/extensions. I think there is potential to use some of these classes outside of just this extension. So that was also part of the reasoning.

I'd like this to be broken down into separate PRs... Can you propose a patch series to make this happen?

Yeah. I honestly didn't realize how big this got until I was preparing the PR. I totally agree. How about:

PR for adding the AWS signer impl and the interfaces of its dependencies.
PR for the AWS IAM credential provider impls and the interface for the HTTP client.
PR for the HTTP client impl. Im definitely interested in using curl if possible.
PR for the actual AWS IAM grpc creds extension

?

htuch · 2019-01-11T00:28:32Z

@htuch

Does source/common/aws belong in extensions instead, given this is a gRPC credential extension?

This is a result of the earlier version of this code. We had to make changes to Main to initialize the AWS SDK. Easy enough to move to source/extensions. I think there is potential to use some of these classes outside of just this extension. So that was also part of the reasoning.

There are common directories under source/extensions that might suit this if you want to reuse across extensions, e.g. you could have source/extensions/filters/http/common/aws for cross-extension common code.

I'd like this to be broken down into separate PRs... Can you propose a patch series to make this happen?

Yeah. I honestly didn't realize how big this got until I was preparing the PR. I totally agree. How about:

PR for adding the AWS signer impl and the interfaces of its dependencies.

PR for the AWS IAM credential provider impls and the interface for the HTTP client.

PR for the HTTP client impl. Im definitely interested in using curl if possible.

PR for the actual AWS IAM grpc creds extension

?

This seems a reasonable breakdown, thanks.

As an aside, I'm curious if you folks through about making the Envoy independent parts a standalone library that other projects could consume (who don't need the full AWS SDK). The upside if you do that is you don't need us to review that code and get more reuse ;) OTOH, it seems reasonably scoped for an extension if it's not going to grow a ton, so happy to review as is.

As part of leveraging Envoy in AWS App Mesh, we implemented a gRPC credentials extension that uses AWS IAM auth. To be able to use authenticate gRPC requests with AWS IAM credentials, we needed to add a code to Envoy to sign HTTP requests. More high-level details in #5215. I opened up a prior PR: #5546 which was admittedly too large and unwieldy. This is part 1 of a series of smaller patches needed to add our gRPC authentication extension to Envoy. Risk Level: Low Testing: Unit tests for this API. This has also been verified to work when communicating with App Mesh. Signed-off-by: Scott LaVigne <lavignes@amazon.com>

As part of leveraging Envoy in AWS App Mesh, we implemented a gRPC credentials extension that uses AWS IAM auth. To be able to use authenticate gRPC requests with AWS IAM credentials, we needed to add a code to Envoy to sign HTTP requests. More high-level details in envoyproxy#5215. I opened up a prior PR: envoyproxy#5546 which was admittedly too large and unwieldy. This is part 1 of a series of smaller patches needed to add our gRPC authentication extension to Envoy. Risk Level: Low Testing: Unit tests for this API. This has also been verified to work when communicating with App Mesh. Signed-off-by: Scott LaVigne <lavignes@amazon.com> Signed-off-by: Dan Zhang <danzh@google.com>

As part of leveraging Envoy in AWS App Mesh, we implemented a gRPC credentials extension that uses AWS IAM auth. To be able to use authenticate gRPC requests with AWS IAM credentials, we needed to add a code to Envoy to sign HTTP requests. More high-level details in envoyproxy#5215. I opened up a prior PR: envoyproxy#5546 which was admittedly too large and unwieldy. This is part 1 of a series of smaller patches needed to add our gRPC authentication extension to Envoy. Risk Level: Low Testing: Unit tests for this API. This has also been verified to work when communicating with App Mesh. Signed-off-by: Scott LaVigne <lavignes@amazon.com> Signed-off-by: Scott LaVigne <1406859+lavignes@users.noreply.github.com>

As part of leveraging Envoy in AWS App Mesh, we implemented a gRPC credentials extension that uses AWS IAM auth. To be able to use authenticate gRPC requests with AWS IAM credentials, we needed to add a code to Envoy to sign HTTP requests. More high-level details in envoyproxy#5215. I opened up a prior PR: envoyproxy#5546 which was admittedly too large and unwieldy. This is part 1 of a series of smaller patches needed to add our gRPC authentication extension to Envoy. Risk Level: Low Testing: Unit tests for this API. This has also been verified to work when communicating with App Mesh. Signed-off-by: Scott LaVigne <lavignes@amazon.com> Signed-off-by: Fred Douglas <fredlas@google.com>

lavignes added 8 commits January 8, 2019 14:36

grpc: add context to grpc credentials factory

561b1fd

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

Add AWS Sigv4 signer

7f1d1e1

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

grpc: Add AWS IAM grpc credentials extension

7d3727d

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

docs: added version-history line

35e84a1

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

Add external dep on signer

b53c163

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

grpc: Fix google async client tests

d18c246

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

Fix visibility for aws auth tests

e5a09f6

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

Potentially fix macOS build

f57e081

Signed-off-by: Scott LaVigne <lavignes@amazon.com>

htuch self-assigned this Jan 9, 2019

lavignes mentioned this pull request Jan 12, 2019

Add AWS http request signer #5580

Merged

lavignes closed this Jan 14, 2019

lavignes mentioned this pull request Feb 13, 2019

xDS gRPC credentials for AWS IAM #5215

Closed

yuval-k mentioned this pull request Apr 16, 2019

support for AWS upstream with secure IAM role access kgateway-dev/kgateway#680

Closed

ivitjuk mentioned this pull request Jun 5, 2019

Added AWS region provider #7172

Merged

ivitjuk mentioned this pull request Jul 8, 2019

Aws sigv4 metadata fetcher #7303

Merged

lavignes mentioned this pull request Jan 20, 2020

proxying requests to AWS authenticated services (e.g.: s3) #9708

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

grpc: Add AWS IAM grpc credentials extension #5546

grpc: Add AWS IAM grpc credentials extension #5546

Uh oh!

lavignes commented Jan 8, 2019

Uh oh!

htuch commented Jan 9, 2019

Uh oh!

htuch commented Jan 9, 2019

Uh oh!

ggreenway commented Jan 9, 2019

Uh oh!

lavignes commented Jan 10, 2019

Uh oh!

htuch commented Jan 11, 2019

Uh oh!

Uh oh!

grpc: Add AWS IAM grpc credentials extension #5546

grpc: Add AWS IAM grpc credentials extension #5546

Uh oh!

Conversation

lavignes commented Jan 8, 2019

Uh oh!

htuch commented Jan 9, 2019

Uh oh!

htuch commented Jan 9, 2019

Uh oh!

ggreenway commented Jan 9, 2019

Uh oh!

lavignes commented Jan 10, 2019

Uh oh!

htuch commented Jan 11, 2019

Uh oh!

Uh oh!