Blog for secure webhook management #5285

lei-tang · 2019-10-29T19:34:19Z

This PR has URLs pointing to: #5162.
Thus the lint test will fail until these PRs are merged.

Please provide a description for what this PR is for: istio/istio#17061.

Design documents:
[1]https://docs.google.com/document/d/1VHyBre8943USOx_YEVtA18KfEoGTmLT1iBHESO5bzcA/edit#heading=h.qex63c29z2to
[2] https://docs.google.com/document/d/1v8BxI07u-mby5f5rCruwF7odSXgb9G8-C9W5hQtSIAg/edit#heading=h.xw1gqgyqs5b

And to help us figure out who should review this PR, please
put an X in all the areas that this PR affects.

[ ] Configuration Infrastructure
[ ] Docs
[ X] Installation
[ X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

ericvn · 2019-10-31T15:30:21Z

/test lint_istio.io

adammil2000

A few comments here, but looks great! Thanks!

adammil2000 · 2019-10-31T18:14:04Z

content/en/blog/2019/webhook/index.md

+target_release: 1.4
+---
+
+Istio has two webhooks: Galley and Sidecar Injector. Both of them are critical to Istio,


Suggested change

Istio has two webhooks: Galley and Sidecar Injector. Both of them are critical to Istio,

Istio has two webhooks: Galley and Sidecar Injector. Both of them are crucial for Istio,

Done, thank you!

adammil2000 · 2019-10-31T18:14:31Z

content/en/blog/2019/webhook/index.md

+    caption="An example attack"
+    >}}
+
+In Istio 1.4, we introduce a new feature that uses `istioctl` to securely manage webhooks, with which:


Suggested change

In Istio 1.4, we introduce a new feature that uses `istioctl` to securely manage webhooks, with which:

To protect against this kind of attack, Istio 1.4 introduces a new feature to securely manage

webhooks using `istioctl`:

Done, thank you!

adammil2000

LGTM thanks!

jwendell · 2019-11-02T15:37:47Z

@lei-tang Could you please verify the lint failure?

lei-tang · 2019-11-02T18:09:59Z

@jwendell This PR has URLs pointing to: #5162. Thus the lint test will fail until the PR 5162 is merged.

rcaballeromx

Thanks for the blog post announcement for the feature.
However, the text is very difficult to follow and contains forward looking statements.
@adammil2000 Could you help @lei-tang address some of the issues still present?

rcaballeromx · 2019-11-03T20:34:05Z

content/en/blog/2019/webhook/index.md

+target_release: 1.4
+---
+
+Istio has two webhooks: Galley and Sidecar Injector. Both of them are crucial for Istio,


Do not capitalize the sidecar injector unless you are referring specifically to the value in a config file.

Suggested change

Istio has two webhooks: Galley and Sidecar Injector. Both of them are crucial for Istio,

Istio has two webhooks: Galley and the sidecar injector. Both of them are crucial for Istio,

Revised accordingly.

rcaballeromx · 2019-11-03T20:35:44Z

content/en/blog/2019/webhook/index.md

+sidecar containers.
+
+By default, Galley and Sidecar Injector manage their own webhook configurations, which may
+have a security risk if they are compromised (e.g., by buffer overflow attacks). Configuring


Suggested change

have a security risk if they are compromised (e.g., by buffer overflow attacks). Configuring

have a security risk if they are compromised, for example, through buffer overflow attacks. Configuring

Revised accordingly.

rcaballeromx · 2019-11-03T20:43:57Z

content/en/blog/2019/webhook/index.md

+To protect against this kind of attack, Istio 1.4 introduces a new feature to securely manage
+webhooks using `istioctl`:
+
+1. `istioctl`, instead of Galley and Sidecar Injector, manage the webhook configurations.


This paragraph is very confusing. @lei-tang What are the features introduced? You are not talking about istioctl but rather the ability to use istioctl to perform certain operations. Please revisit the text on this list and mke it clear to the readers the following:

What new operations they can perform using istioctl?

What are the new sub-commands or flags?

The feature introduced is istioctl, instead of Galley and the sidecar injector, manage the webhook configurations. The sentences here highlights the advantages of the new feature; the details of the new feature is in the user guide.

rcaballeromx · 2019-11-03T20:45:07Z

content/en/blog/2019/webhook/index.md

+and whether the certificate chain used by the webhook server is valid, which reduces the errors
+caused by an unready server and by invalid certificates.
+
+To try this new feature, please follow its [user guide](/docs/setup/install/webhook).


Do not introduce the content for this functionality as part of our setup guide. The content should be within /docs/tasks/security/

Revised accordingly.

zjory · 2019-11-07T23:35:09Z

content/en/blog/2019/webhook/index.md

+
+Istio has two webhooks: Galley and the sidecar injector. Both of them are crucial for Istio,
+with Galley validating Kubernetes resources and the sidecar injector injecting Istio
+sidecar containers.


Istio has two webhooks: Galley and the sidecar injector. Both of them are crucial for Istio,
with Galley validates Kubernetes resources and the sidecar injector injects~~ing~~ sidecar containers into Istio
~~sidecar containers~~.

Revised accordingly.

zjory · 2019-11-07T23:37:45Z

content/en/blog/2019/webhook/index.md

+sidecar containers.
+
+By default, Galley and the sidecar injector manage their own webhook configurations, which may
+have a security risk if they are compromised, for example, through buffer overflow attacks. Configuring


By default, Galley and the sidecar injector manage their own webhook configurations.,which may
have This can pose a security risk if they are compromised, for example, through buffer overflow attacks. Configuring

Revised accordingly.

zjory · 2019-11-07T23:38:50Z

content/en/blog/2019/webhook/index.md

+have a security risk if they are compromised, for example, through buffer overflow attacks. Configuring
+a webhook is a highly privileged operation as a webhook may monitor and mutate all
+Kubernetes resources. In the following example, the attacker compromises
+Galley and modifies the webhook configuration of Galley to eavesdrop all Kubernetes secrets


lley and modifies the webhook configuration of Galley to eavesdrop on all Kubernetes secrets

Revised accordingly.

zjory · 2019-11-07T23:39:20Z

content/en/blog/2019/webhook/index.md

+Kubernetes resources. In the following example, the attacker compromises
+Galley and modifies the webhook configuration of Galley to eavesdrop all Kubernetes secrets
+(the `clientConfig` is modified by the attacker to direct the `secrets` resources to the
+a service owned by the attacker).


(the clientConfig is modified by the attacker to direct the secrets resources to ~~the~~
a service owned by the attacker).

Revised accordingly.

zjory · 2019-11-07T23:40:18Z

content/en/blog/2019/webhook/index.md

+webhooks using `istioctl`:
+
+1. `istioctl`, instead of Galley and the sidecar injector, manage the webhook configurations.
+Galley and the sidecar injector are de-privileged such that even if they are compromised, they


Galley and the sidecar injector are de-privileged ~~such that~~ so even if they are compromised, they

Revised accordingly.

zjory · 2019-11-07T23:41:01Z

content/en/blog/2019/webhook/index.md

+will not be able to alter the webhook configurations.
+
+1. Before configuring a webhook, `istioctl` will verify whether the webhook server is up
+and whether the certificate chain used by the webhook server is valid, which reduces the errors


Before configuring a webhook, istioctl will verify ~~whether~~ the webhook server is up
and ~~whether~~ that the certificate chain used by the webhook server is valid, which reduces the errors

Revised accordingly.

zjory

@rcaballeromx Completed content review

frankbu · 2019-11-08T15:59:04Z

content/en/blog/2019/webhook/index.md

+---
+title: Secure webhook management
+description: A more secure way to manage Istio webhooks.
+publishdate: 2019-10-29


Should the date of both of your blogs be changed to align with the release date?

frankbu · 2019-11-08T16:00:33Z

content/en/blog/2019/webhook/index.md

+will not be able to alter the webhook configurations.
+
+1. Before configuring a webhook, `istioctl` will verify the webhook server is up
+and that the certificate chain used by the webhook server is valid, which reduces the errors


Suggested change

and that the certificate chain used by the webhook server is valid, which reduces the errors

and that the certificate chain used by the webhook server is valid. This reduces the errors

frankbu · 2019-11-08T16:06:05Z

content/en/blog/2019/webhook/index.md

+
+1. Before configuring a webhook, `istioctl` will verify the webhook server is up
+and that the certificate chain used by the webhook server is valid, which reduces the errors
+caused by an unready server and by invalid certificates.


Not sure what this is saying? Maybe:

Suggested change

caused by an unready server and by invalid certificates.

that can occur before a server is ready or has invalid certificates.

frankbu · 2019-11-08T16:06:40Z

content/en/blog/2019/webhook/index.md

+and that the certificate chain used by the webhook server is valid, which reduces the errors
+caused by an unready server and by invalid certificates.
+
+To try this new feature, please follow its [user guide](/docs/setup/security/webhook).


Suggested change

To try this new feature, please follow its [user guide](/docs/setup/security/webhook).

To try this new feature, refer to the [Istio webhook management task](/docs/setup/security/webhook).

frankbu · 2019-11-08T16:07:19Z

content/en/blog/2019/webhook/index.md

+By default, Galley and the sidecar injector manage their own webhook configurations.
+This can pose a security risk if they are compromised, for example, through buffer overflow attacks.
+Configuring a webhook is a highly privileged operation as a webhook may monitor and mutate all
+Kubernetes resources. In the following example, the attacker compromises


Suggested change

Kubernetes resources. In the following example, the attacker compromises

Kubernetes resources.

In the following example, an attacker compromises

lei-tang · 2019-11-08T16:15:05Z

@frankbu The PR has been revised based on the comments. Please take another look. Thank you!

lei-tang · 2019-11-08T16:28:24Z

/test lint_istio.io

frankbu · 2019-11-08T16:29:55Z

content/en/blog/2019/webhook/index.md

+
+1. Before configuring a webhook, `istioctl` will verify the webhook server is up
+and that the certificate chain used by the webhook server is valid. This reduces the errors
+that can occur before a server is ready or has invalid certificates.


Suggested change

that can occur before a server is ready or has invalid certificates.

that can occur before a server is ready or if a server has invalid certificates.

frankbu

Please make one more suggested tweak to the wording.

All addressed.

lei-tang requested a review from a team as a code owner October 29, 2019 19:34

istio-policy-bot added area/environments area/networking area/security labels Oct 29, 2019

googlebot added the cla: yes Set by the Google CLA bot to indicate the author of a PR has signed the Google CLA. label Oct 29, 2019

istio-testing added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 29, 2019

lei-tang requested review from rlenglet, myidpt, costinm and ostromart October 29, 2019 19:35

lei-tang added this to the 1.4 milestone Oct 30, 2019

rlenglet approved these changes Oct 30, 2019

View reviewed changes

adammil2000 reviewed Oct 31, 2019

View reviewed changes

adammil2000 approved these changes Oct 31, 2019

View reviewed changes

rcaballeromx previously requested changes Nov 3, 2019

View reviewed changes

lei-tang force-pushed the blog-webhook-controller branch from 4864147 to a0786be Compare November 4, 2019 18:40

lei-tang requested a review from rcaballeromx November 4, 2019 18:43

lei-tang mentioned this pull request Nov 4, 2019

User guide for istioctl managing webhooks #5162

Merged

zjory reviewed Nov 7, 2019

View reviewed changes

frankbu reviewed Nov 8, 2019

View reviewed changes

frankbu approved these changes Nov 8, 2019

View reviewed changes

lei-tang added 6 commits November 8, 2019 08:37

Blog for secure webhook management

5767f96

Revise the wording

c1eaf6d

Revise wording and location of guide

028ba32

Revise the writing

2fdc1c0

Revised based on review comments

ea235f9

Fix a wording

1ac1588

lei-tang force-pushed the blog-webhook-controller branch from 3f04a3d to 1ac1588 Compare November 8, 2019 16:37

Fix a link

f51357f

istio-testing merged commit c2c00f5 into istio:master Nov 8, 2019

	Istio has two webhooks: Galley and Sidecar Injector. Both of them are critical to Istio,
	Istio has two webhooks: Galley and Sidecar Injector. Both of them are crucial for Istio,

	In Istio 1.4, we introduce a new feature that uses `istioctl` to securely manage webhooks, with which:
	To protect against this kind of attack, Istio 1.4 introduces a new feature to securely manage
	webhooks using `istioctl`:

	have a security risk if they are compromised (e.g., by buffer overflow attacks). Configuring
	have a security risk if they are compromised, for example, through buffer overflow attacks. Configuring

	and that the certificate chain used by the webhook server is valid, which reduces the errors
	and that the certificate chain used by the webhook server is valid. This reduces the errors

	caused by an unready server and by invalid certificates.
	that can occur before a server is ready or has invalid certificates.

	To try this new feature, please follow its [user guide](/docs/setup/security/webhook).
	To try this new feature, refer to the [Istio webhook management task](/docs/setup/security/webhook).

Blog for secure webhook management #5285

Blog for secure webhook management #5285

Uh oh!

Conversation

lei-tang commented Oct 29, 2019

Uh oh!

ericvn commented Oct 31, 2019

Uh oh!

adammil2000 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adammil2000 left a comment

Choose a reason for hiding this comment

Uh oh!

jwendell commented Nov 2, 2019

Uh oh!

lei-tang commented Nov 2, 2019

Uh oh!

rcaballeromx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjory left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!