add e2e test for galley #5047

ayj · 2018-04-18T18:47:17Z

This currently tests the configuration validation service. Additional test cases will be added overtime to cover other aspects of Galley.

xiaolanz · 2018-04-18T20:25:05Z

install/updateVersion.sh

@@ -55,8 +55,11 @@ function gen_file() {
    fi
 }

+TARGETS="istio.yaml istio-auth.yaml istio-one-namespace.yaml istio-one-namespace-auth.yaml"


in go, you don't need to split lines.

I split the lines for readability. Alternatively, it could be written with line continuations (see below), but this seemed less error prone. Also, this is a bash script :)

TARGETS="istio.yaml istio-auth.yaml istio-one-namespace.yaml istio-one-namespace-auth.yaml \ istio-multicluster.yaml istio-auth-multicluster.yaml istio-remote.yaml \ istio-galley.yaml istio-auth-galley.yaml"

xiaolanz · 2018-04-18T20:28:24Z

tests/e2e/tests/galley/testdata/mixer-handler.yaml

@@ -0,0 +1,9 @@
+apiVersion: "config.istio.io/v1alpha2"


This adds burden to update tests files when config apis changes.
is there a way that we can reuse existing yamls to reduce test maintenance cost?

We'll need valid and invalid variants of each group/version/kind. And I don't think we have that exhaustive list in istio/istio. I figured it was better to be consistent and define them in one testdata location.

GVK shouldn't change that frequently either (post config group rename churn) and the total list of mixer kinds should be reduced by an order of magnitude once mixer config is refactored into common instance and handler kinds.

ayj · 2018-04-18T20:42:22Z

tests/e2e/tests/galley/galley_test.go

+	}
+	tc = &testConfig{CommonConfig: cc}
+
+	// TODO - enforce test constraints


@ozevren, checking flags is less than ideal but does succeed in guarding the test from running in the wrong environment. Ideally this could be written as test.Required("galley", "cluster-wide") and the test framework would ensure galley is running in a cluster-wide environment.

codecov · 2018-04-19T17:44:42Z

Codecov Report

Merging #5047 into master will decrease coverage by 1%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #5047    +/-   ##
=======================================
- Coverage      74%     73%   -<1%     
=======================================
  Files         323     319     -4     
  Lines       27134   26553   -581     
=======================================
- Hits        19913   19330   -583     
- Misses       6440    6443     +3     
+ Partials      781     780     -1

Impacted Files	Coverage Δ
pilot/pkg/model/jwks_resolver.go	`56% <0%> (-21%)`	⬇️
mixer/adapter/stackdriver/helper/common.go	`71% <0%> (-12%)`	⬇️
mixer/adapter/solarwinds/metrics_handler.go	`75% <0%> (-8%)`	⬇️
istioctl/cmd/istioctl/convert/cmd.go	`40% <0%> (-3%)`	⬇️
mixer/adapter/opa/opa.go	`79% <0%> (-3%)`	⬇️
pilot/pkg/model/egress_rules.go	`95% <0%> (-2%)`	⬇️
...ilot/pkg/networking/plugin/authn/authentication.go	`67% <0%> (ø)`	⬇️
pilot/pkg/serviceregistry/aggregate/controller.go	`80% <0%> (ø)`	⬇️
mixer/adapter/stackdriver/metric/metric.go	`87% <0%> (ø)`	⬇️
pilot/pkg/model/conversion.go	`89% <0%> (ø)`	⬇️
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fe027a...bbabfbf. Read the comment docs.

ayj · 2018-04-19T18:15:54Z

/cc @costinm for circleci changes

/cc @sebastienvas for test and prow changes

/cc @yusuoh for install updates. Also, this PR overlaps a bit with #4926 in how it handle the YAML file selection.

yusuoh · 2018-04-19T18:26:15Z

tests/e2e/framework/kubernetes.go

 	} else {
 		if *authEnable {
 			istioYaml = authInstallFileNamespace
 			if *multiClusterDir != "" {
 				istioYaml = mcAuthInstallFileNamespace
 			}
 		}
+		if *useGalleyConfigValidator {


Maybe we should also try to merge all these logic into the helper function and call it getInstallFile()

Well - we should remove all this logic and the config file ASAP. We have it just because the helm migration is not complete.

The future is to use help template directly, and pass any customization as flags - either by doing the install of istio in a separate target before the e2e test, or by passing helm options to the install (so we test what users will really use, not the deprecated files)

I think this is the best we can do in the current PR. Refactoring the test framework to invoke helm directly is a larger effort and outside the scope of this task.

xiaolanz

/lgtm

…cleci

ayj · 2018-04-25T16:19:27Z

Well - we should remove all this logic and the config file ASAP. We have it just because the helm migration is not complete. The future is to use help template directly, and pass any customization as flags - either by doing the install of istio in a separate target before the e2e test, or by passing helm options to the install (so we test what users will really use, not the deprecated files)

Agreed. And I believe this is being addressed by the testing SWAT team's efforts.

ayj · 2018-04-25T18:26:28Z

2018/04/25 01:45:12 ci2gubernator.go:103: Generating finished.json
2018/04/25 01:45:12 ci2gubernator.go:109: Uploading finished.json to gs://istio-circleci/e2e-galley/80582/finished.json
2018/04/25 01:45:13 ci2gubernator.go:126: Uploading build-log.txt to gs://istio-circleci/e2e-galley/80582/build-log.txt
2018/04/25 01:45:13 ci2gubernator.go:191: Uploading /go/out/tests/junit.xml to gs://istio-circleci/e2e-galley/80582/artifacts/junit.xml
2018/04/25 01:45:14 gcsClient.go:57: Failed to open a reader on file istio-circleci/e2e-galley/latest-build.txt from gcs, storage: object doesn't exist
2018/04/25 01:45:15 main.go:92: Failed to update latest-build.txt to 80582: storage: object doesn't exist

@chxchx- FYI, I manually created an empty gs://istio-circleci/e2e-galley/latest-build.txt and re-ran the test. Let me know if there is something else I should to fix this error.

ayj · 2018-04-25T18:35:41Z

@costinm, any other remaining concerns with the PR? The comments re: general testing methodology, use of helm,etc are being addressed by testing SWAT team.

ayj · 2018-04-26T20:00:21Z

/test istio-pilot-e2e

istio-testing · 2018-04-26T20:10:53Z

@ayj: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
prow/e2e-bookInfoTests-v1alpha3.sh	`bbabfbf`	link	`/test e2e-bookInfo-envoyv2-v1alpha3`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

xiaolanz · 2018-04-27T16:25:34Z

/lgtm
/approval

yusuoh · 2018-04-27T16:27:47Z

/lgtm

istio-testing · 2018-04-27T16:27:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xiaolanz, yusuoh
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm

Assign the PR to them by writing /assign @costinm in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

istio-testing · 2018-05-11T22:24:40Z

@ayj: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ayj · 2018-05-22T16:27:10Z

@rshriram, can you re-open this PR? I don't seem to have permissions to do it myself.

The PR itself isn't stale. Master has been frozen so I hadn't been rebasing.

sebastienvas · 2018-05-22T16:56:14Z

/reopen

istio-testing · 2018-05-22T16:56:14Z

@sebastienvas: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ozevren · 2018-05-22T17:07:22Z

/reopen

istio-testing · 2018-05-22T17:07:22Z

@ozevren: The reopen prow plugin is deprecated, please migrate to the lifecycle plugin before April 2018

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

istio-testing · 2018-05-22T17:07:24Z

@ozevren: failed to re-open PR: state cannot be changed. The add-galley-e2e-test branch was force-pushed or recreated.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ayj · 2018-05-22T17:12:17Z

/reopen

istio-testing · 2018-05-22T17:12:18Z

@ayj: The reopen prow plugin is deprecated, please migrate to the lifecycle plugin before April 2018

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

istio-testing · 2018-05-22T17:12:19Z

@ayj: failed to re-open PR: state cannot be changed. The add-galley-e2e-test branch was force-pushed or recreated.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wip: add e2e galley validation test

5dab0ce

ayj requested review from ozevren and xiaolanz April 18, 2018 18:47

googlebot added the cla: yes label Apr 18, 2018

ayj added the area/config label Apr 18, 2018

xiaolanz reviewed Apr 18, 2018

View reviewed changes

ayj commented Apr 18, 2018

View reviewed changes

fill in test data

0eb0820

ayj changed the title ~~wip: add e2e test for galley validation~~ add e2e test for galley validation Apr 19, 2018

ayj added 2 commits April 19, 2018 11:06

enable galley validation test in e2e cluster wide auth test suite

8cb5a99

enable e2e_galley in circleci

d41e648

ayj changed the title ~~add e2e test for galley validation~~ add e2e test for galley Apr 19, 2018

istio-testing requested review from sebastienvas and costinm April 19, 2018 18:15

istio deleted a comment from istio-testing Apr 19, 2018

ayj requested review from yusuoh and costinm and removed request for costinm April 19, 2018 18:17

fix linter

caec8b0

yusuoh reviewed Apr 19, 2018

View reviewed changes

xiaolanz reviewed Apr 19, 2018

View reviewed changes

istio-testing assigned xiaolanz Apr 19, 2018

istio-testing added the lgtm label Apr 19, 2018

ayj added 2 commits April 19, 2018 11:52

more linter fixes

e549c14

enable MutatingAdmissionWebhook and ValidatingAdmissionWebhook in cir…

834ea67

…cleci

istio-testing added needs-rebase Indicates a PR needs to be rebased before being merged and removed lgtm labels Apr 19, 2018

Merge branch 'master' of github.com:istio/istio into add-galley-e2e-test

5b249eb

ayj removed the request for review from sebastienvas April 26, 2018 19:00

Merge branch 'master' into add-galley-e2e-test

bbabfbf

istio-testing added the lgtm label Apr 27, 2018

istio-testing assigned yusuoh Apr 27, 2018

yusuoh approved these changes Apr 27, 2018

View reviewed changes

istio-testing added the needs-rebase Indicates a PR needs to be rebased before being merged label May 11, 2018

rshriram added the stale? label May 22, 2018

rshriram closed this May 22, 2018

ayj assigned ozevren and unassigned xiaolanz and yusuoh May 22, 2018

ayj mentioned this pull request May 22, 2018

add galley e2e test #5757

Merged

add e2e test for galley #5047

add e2e test for galley #5047

Uh oh!

Conversation

ayj commented Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaolanz Apr 18, 2018

Choose a reason for hiding this comment

Uh oh!

ayj Apr 18, 2018

Choose a reason for hiding this comment

Uh oh!

xiaolanz Apr 18, 2018

Choose a reason for hiding this comment

Uh oh!

ayj Apr 18, 2018

Choose a reason for hiding this comment

Uh oh!

ayj Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Apr 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ayj commented Apr 19, 2018

Uh oh!

yusuoh Apr 19, 2018

Choose a reason for hiding this comment

Uh oh!

costinm Apr 25, 2018

Choose a reason for hiding this comment

Uh oh!

ayj Apr 25, 2018

Choose a reason for hiding this comment

Uh oh!

xiaolanz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayj commented Apr 25, 2018

Uh oh!

ayj commented Apr 25, 2018

Uh oh!

ayj commented Apr 25, 2018

Uh oh!

ayj commented Apr 26, 2018

Uh oh!

istio-testing commented Apr 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaolanz commented Apr 27, 2018

Uh oh!

yusuoh commented Apr 27, 2018

Uh oh!

istio-testing commented Apr 27, 2018

Uh oh!

istio-testing commented May 11, 2018

Uh oh!

ayj commented May 22, 2018

Uh oh!

sebastienvas commented May 22, 2018

Uh oh!

istio-testing commented May 22, 2018

Uh oh!

ozevren commented May 22, 2018

Uh oh!

istio-testing commented May 22, 2018

Uh oh!

istio-testing commented May 22, 2018

Uh oh!

ayj commented May 22, 2018

Uh oh!

istio-testing commented May 22, 2018

Uh oh!

istio-testing commented May 22, 2018

Uh oh!

Uh oh!

ayj commented Apr 18, 2018 •

edited

Loading

ayj Apr 18, 2018 •

edited

Loading

codecov bot commented Apr 19, 2018 •

edited

Loading

xiaolanz left a comment •

edited

Loading

istio-testing commented Apr 26, 2018 •

edited

Loading