Skip to content

Conversation

andraxylia
Copy link
Contributor

@andraxylia andraxylia commented Mar 15, 2018

Upgrade to new proxy to pick up Envoy cluster warming.

New Envoy also works around this failure:
[2018-03-15 18:13:33.252][10][warning][upstream] external/envoy/source/server/lds_subscription.cc:68] lds: fetch failure: gRPC client cluster 'mixer_check_server' is not static

Update dependency in Makefile, otherwise the build does not pick up the new Envoy (some recent changes were half-baked) and proxy upgrade did not work.

@andraxylia andraxylia requested a review from a team March 15, 2018 17:03
@mandarjog
Copy link
Contributor

/lgtm

@istio-merge-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mandarjog

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

Copy link
Member

@ldemailly ldemailly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it the only place we have that sha now? (did you grep for the old one)

@codecov
Copy link

codecov bot commented Mar 15, 2018

Codecov Report

Merging #4292 into master will decrease coverage by 1%.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #4292    +/-   ##
=======================================
- Coverage      71%     71%   -<1%     
=======================================
  Files         315     315            
  Lines       28616   29094   +478     
=======================================
+ Hits        20116   20378   +262     
- Misses       7329    7491   +162     
- Partials     1171    1225    +54
Impacted Files Coverage Δ
mixer/adapter/servicecontrol/distValueBuilder.go 88% <0%> (-4%) ⬇️
mixer/adapter/denier/denier.go 89% <0%> (-3%) ⬇️
mixer/adapter/opa/opa.go 79% <0%> (-3%) ⬇️
...olarwinds/internal/papertrail/papertrail_logger.go 59% <0%> (-3%) ⬇️
mixer/adapter/statsd/statsd.go 96% <0%> (-1%) ⬇️
mixer/adapter/rbac/rbacStore.go 83% <0%> (-1%) ⬇️
mixer/template/sample/template.gen.go 55% <0%> (ø) ⬇️
mixer/adapter/circonus/circonus.go 71% <0%> (ø) ⬇️
mixer/adapter/list/ipList.go 100% <0%> (ø) ⬆️
mixer/adapter/stackdriver/metric/merge.go 100% <0%> (ø) ⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e114c0...c077ecf. Read the comment docs.

@istio-merge-robot
Copy link

/lgtm cancel //PR changed after LGTM, removing LGTM. @andraxylia @costinm @mandarjog

@andraxylia andraxylia changed the title Upgrade to new proxy to pick up Envoy cluster warming Upgrade to new proxy to pick up Envoy cluster warming + Makefile fix Mar 15, 2018
@ldemailly
Copy link
Member

superseeds #4086 which does not pass tests either

istio.deps Outdated
@@ -12,6 +12,6 @@
"repoName": "proxy",
"prodBranch": "master",
"file": "",
"lastStableSHA": "80ec4f6c2a672b00e55d34ef9b5d26a42a98da42"
"lastStableSHA": "9b4cbc6559822311d9bc04a6e0dd129b9b75617b"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update this to 9b4cbc6? We would like to include that PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it already is that one :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for correcting me.

@andraxylia
Copy link
Contributor Author

yes, it's the only change that needs to be made to upgrade the proxy. There used to be another one in Docker.proxy, but that one got removed. Can I get a review approved?

@andraxylia
Copy link
Contributor Author

circle-ci e2e pilot fails because of the known issue with routing rule v3.

@kyessenov
Copy link
Contributor

kyessenov commented Mar 15, 2018 via email

@andraxylia
Copy link
Contributor Author

@ZackButcher can you please approve? Need to review faster this kind of PRs.

@costinm costinm self-requested a review March 15, 2018 21:37
@andraxylia
Copy link
Contributor Author

merged with master, but still failing with v3, so that unrelated issue is not fixed.

@kyessenov
Copy link
Contributor

kyessenov commented Mar 15, 2018 via email

@andraxylia
Copy link
Contributor Author

I am sure I merged to master.

@andraxylia
Copy link
Contributor Author

/retest all

@andraxylia
Copy link
Contributor Author

/test istio-presubmit

@kyessenov
Copy link
Contributor

kyessenov commented Mar 16, 2018

@andraxylia it failed again in the exact same spot -- it's likely a real failure.

@ldemailly
Copy link
Member

I0316 00:26:09.354] 2018-03-16T00:26:09.354142Z	info	RunBackground: /home/bootstrap/go/src/istio.io/proxy/src/envoy/http/mixer/start_envoy -e /home/bootstrap/go/out/linux_amd64/release/envoy > /tmp/mixer_envoy_env743812681/my_local_proxy.log 2>&1

try
./tests/integration/example/integration.sh
(or
go test -v ./tests/integration/example/tests/sample1 -envoy_binary /home/bootstrap/go/out/linux_amd64/release/envoy -envoy_start_script /home/bootstrap/go/src/istio.io/proxy/src/envoy/http/mixer/start_envoy -mixer_binary /home/bootstrap/go/out/linux_amd64/release/mixs -fortio_binary fortio
)
to run it locally
(@yutongz will improve the script so logs show up in artifacts)

@andraxylia
Copy link
Contributor Author

andraxylia commented Mar 16, 2018

This test is dangerous to run locally. It does a git reset --hard and some git clone.Luckily I did not have anything valuable in my ws.

@andraxylia
Copy link
Contributor Author

Not sure what the value of this test is, and what it is supposed to test. Is it that envoy can be run as a process? It copies envoy back and forth. Until this test is written to a standard where people can understand what the problem is, it adds little value and should be disabled @kyessenov

@andraxylia andraxylia requested a review from a team March 16, 2018 03:54
@andraxylia
Copy link
Contributor Author

andraxylia commented Mar 16, 2018

I run integration.sh manually and debugged it, and this template file from proxy used by the start_envoy script is outdated and does not work with the latest envoy:

https://github.com/istio/proxy/blob/master/src/envoy/http/mixer/envoy.conf.template

See attached output and generated envoy.conf file.

@qiwzhang can help bring that conf file up to date. Meanwhile, the test should be disabled and this PR merged.

envoy.conf.txt
start_envoy_output.txt

@ldemailly
Copy link
Member

Thanks for figuring it out, that's great

Meanwhile, the test should be disabled and this PR merged.

No I think given it works on master it should not be disabled but rather fixed
also we should maybe concerned about the incompatible change

@ldemailly
Copy link
Member

ps: that template should be moved to istio/istio to avoid git cloning that repo just for that 1 file - that part maybe can wait

@andraxylia
Copy link
Contributor Author

Meanwhile, we have to disable this test and remove the "git reset --hard" and all the other dangerous things, including the part where istio/proxy is re-downloaded. Hopefully people will not be tempted to run it locally and loose their changes. Tests should not alter the state of a workspace #4317

https://stackoverflow.com/questions/5788037/recover-from-git-reset-hard/5788069

@andraxylia
Copy link
Contributor Author

@ldemailly this test is not ok, it should be disabled and removed until fixed properly.

@andraxylia
Copy link
Contributor Author

andraxylia commented Mar 16, 2018

That test should also do a make clean before make build, otherwise it uses old envoys from the /home/bootstrap directory. In a shared cluster, this can cause flakiness.

@ldemailly
Copy link
Member

please do file the issues you found with the test, but in the meantime we need the presubmit to pass without dropping the test (imo)

@ldemailly
Copy link
Member

This test is dangerous to run locally. It does a git reset --hard and some git clone.Luckily I did not have anything valuable in my ws.

there is always git reflog but the excerpt I mentioned above I took from the logs:

go test -v ./tests/integration/example/tests/sample1 -envoy_binary /home/bootstrap/go/out/linux_amd64/release/envoy -envoy_start_script /home/bootstrap/go/src/istio.io/proxy/src/envoy/http/mixer/start_envoy -mixer_binary /home/bootstrap/go/out/linux_amd64/release/mixs -fortio_binary fortio

isn't doing doing any of that, the ./tests/integration/example/integration.sh I guess indeed appears to be geared more toward machine than human runs / not to be ran without looking

@andraxylia
Copy link
Contributor Author

That test should not be a pre-submit, there is nothing in it to justify it as a pre-submit.

@sebastienvas
Copy link
Contributor

@yutongz will work on removing the git reset part. I think we should wait until Wayne fixes the config to merge this. The point of this PR is to update Proxy, so if the current proxy is broken, it should not be merged.

@andraxylia
Copy link
Contributor Author

@sebastienvas Wayne has fixed the config and I picked up the fix. I am waiting for the result. The failure of this test does not mean proxy is broken, it only means a file that only Wayne uses is out-of-date. So the test has little value and should not be in istio pre-submit, maybe it should be in istio/proxy presubmit. I suggest we continue the discussion about the usefulness and correctness of that test in #4317 , there are too many things about that test.

@andraxylia
Copy link
Contributor Author

@qiwzhang Did you merge without waiting for all the proxy tests? It appears it did not upload the envoy with that SHA, it is likely the tests that upload it to storage.

@sebastienvas
Copy link
Contributor

@andraxylia SG

@andraxylia
Copy link
Contributor Author

nvm, it seems there is a lag before Envoy is available

@andraxylia
Copy link
Contributor Author

/retest all

@andraxylia andraxylia mentioned this pull request Mar 16, 2018
@andraxylia andraxylia merged commit 8eb6288 into master Mar 18, 2018
@andraxylia andraxylia deleted the cluster_warming branch March 18, 2018 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.