Skip to content

Conversation

sbezverk
Copy link
Contributor

@sbezverk sbezverk commented Apr 5, 2018

Make testEnvLocalK8S.sh script to run on MAC.

  • Fix the script for MAC environment
  • Add deployment of pilot configmap after API server starts
  • Add integration test,
    Starting pilot,
    Creating a service,
    Creating endpoints for new server,
    Check pilot that it has discovered new service

@istio-testing istio-testing added the do-not-merge/hold Block automatic merging of a PR. label Apr 5, 2018
@sbezverk
Copy link
Contributor Author

sbezverk commented Apr 5, 2018

/assign @costinm

@codecov
Copy link

codecov bot commented Apr 5, 2018

Codecov Report

Merging #4773 into master will increase coverage by 1%.
The diff coverage is n/a.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #4773   +/-   ##
======================================
+ Coverage      72%     72%   +1%     
======================================
  Files         302     302           
  Lines       25437   25426   -11     
======================================
+ Hits        18267   18277   +10     
+ Misses       6426    6408   -18     
+ Partials      744     741    -3
Impacted Files Coverage Δ
pilot/pkg/config/memory/monitor.go 82% <0%> (-9%) ⬇️
mixer/adapter/statsd/statsd.go 96% <0%> (-1%) ⬇️
mixer/adapter/rbac/rbacStore.go 84% <0%> (ø) ⬇️
mixer/pkg/protobuf/yaml/resolver.go 100% <0%> (ø) ⬆️
mixer/pkg/adapter/requestData.go 100% <0%> (ø) ⬆️
pilot/pkg/serviceregistry/kube/controller.go 67% <0%> (ø) ⬆️
mixer/adapter/servicecontrol/client.go 0% <0%> (ø) ⬆️
mixer/adapter/memquota/memquota.go 100% <0%> (ø) ⬆️
mixer/adapter/memquota/dedup.go 100% <0%> (ø) ⬆️
pilot/pkg/model/validation.go 85% <0%> (+1%) ⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dcffed...96fe93b. Read the comment docs.

Copy link
Contributor

@costinm costinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great !

@@ -7,10 +7,15 @@ set -euo pipefail
# Based on circleCI config - used to reproduce the environment and to improve local testing

# expect istio scripts to be under $GOPATH/src/istio.io/istio/bin/...
export TOP=$(cd $(dirname $0)/../../../..; pwd)
#export TOP=$(cd $(dirname $0)/../../../..; pwd)
export TOP=${GOPATH}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GOPATH can have multiple components.

But the correct version is ${TOP:-$(...)} - since TOP can be set beforehand when running, it is set by makefile.

@@ -128,49 +133,82 @@ function startLocalApiserver() {
# make sure apiserver is actually alive
kill -0 $(cat $LOG_DIR/apiserver.pid)

echo "Started local etcd and apiserver !"
# Really need to make sure that API Server is up before proceed further
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it's a separate call because the tests scripts (at least in circle) can do some other work in parallel, while this is starting.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However the startup is now fast enough - so maybe it doesn't matter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@costinm actually I added this function because I hit a failure scenario when I was trying to create a configmap but API was not ready for it. I noticed at least 3 seconds delay before API server is ready to accept requests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can leave it the way it was - start, do other things (build docker images in case of CI), waitForApiserver, apply config maps.

But we can refine it - 3 seconds is not that bad.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not waiting for a reliable indication that api server is up and running can potentially introduced flakiness to tests. IMHO we should relay on confirmed state before moving forward.

kubectl get nodes 2>/dev/null || startLocalApiserver
}

function createIstioConfigmap() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general - we're trying to save the output in $OUT_DIR, not tmp.

Maybe we can do a bit more - and apply the entire istio setup ? (of course no pod will be started).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you guys are mounting config files into pods, so it will not be possible in local environment. I see two possibility, 1 - refactor the way a process gets configuration and instead of mounting it inside of a container, directly get config information (client-go api) from a configmap and unmarshal it into in memory structs, 2 - extract config into files and place them at location the process expect to see it. It is doable if the location of config for a process is customizable, not hard coded like /etc/pilot/config for example.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, location should not be hardcoded - we can add some env variable to control it (PILOT_CONFIG ?).

We may also do (1) - and watch so we handle config changes, but (2) should be done either way, we also support
cases where there is no k8s.

echo $! > $LOG_DIR/envoy4.pid
echo $! > $LOG_DIR/envoy4.pid
else
printf "No Envoy on MAC, sorry...\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently there is one. Piotr got it to compile - he needs to upload it somewhere until we get auto-builds working.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great, looking forward for its location.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PiotrSikora could you upload envoy for mac container somewhere so this PR work can continue? I don't use mac, but alot of folks want to do so, and it would be appreciated.

Cheers
-steve

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sdake there is no such thing as "mac container", only macOS binary.

@hklai @sebastienvas can we get official builds of Istio and Istio Proxy for macOS? I'd rather not have unofficial builds pulled as part of official tools.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PiotrSikora Agree about official thing, but if you could get me unofficial just so I could continue playing with it, that would be very nice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PiotrSikora I don't use Mac for dev, I think @sbezverk does - and perhaps that was confusing about this request.

Copy link
Contributor

@PiotrSikora PiotrSikora Apr 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooops, indeed, I got confused by the fairly similar usernames, sorry about that!

Anyway, I pushed Istio Proxy built on macOS from 0.7.1 release to https://storage.googleapis.com/istio-on-macos/releases/0.7.1/istio-proxy-0.7.1-macos.tar.gz.

If you're on macOS, then you can build it with:

$ git checkout 0.7.1
$ bazel build --config=release //src/envoy:envoy

and verify that it matches binary in the linked archive:

$ openssl sha256 bazel-bin/src/envoy/envoy
SHA256(bazel-bin/src/envoy/envoy)= 8dbe0adab7761d61ba3a531531e86922a2106567fe1196c638a65fec16a66627
$ curl -Os https://storage.googleapis.com/istio-on-macos/releases/0.7.1/istio-proxy-0.7.1-macos.tar.gz
$ tar -xOf istio-proxy-0.7.1-macos.tar.gz | openssl sha256
(stdin)= 8dbe0adab7761d61ba3a531531e86922a2106567fe1196c638a65fec16a66627

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PiotrSikora @sdake

We don't build mac binaries in release, so I don't suppose we will have any official macos binaries.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hklai Fair. I believe @sbezverk was asking for unofficial builds so he could make forward progress on this PR. Not sure though how other people would consume this PR without official Mac builds of proxy.

Cheers
-steve

Copy link
Contributor Author

@sbezverk sbezverk Apr 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PiotrSikora @costinm Thanks a lot for the link, when I try to start envoy using envoy_local.json, I see the following errors:

Envoy starting...
[2018-04-10 11:01:55.957][12685001][info][main] external/envoy/source/server/server.cc:178] initializing epoch 0 (hot restart version=disabled)
[2018-04-10 11:01:55.962][12685001][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:127] cm init: initializing cds
[2018-04-10 11:01:55.963][12685001][info][config] external/envoy/source/server/configuration_impl.cc:52] loading 6 listener(s)
[2018-04-10 11:01:55.965][12685001][critical][main] external/envoy/source/server/server.cc:71] error initializing configuration 'tests/testdata/envoy_local.json': route: unknown cluster 'service1'
[2018-04-10 11:01:55.965][12685001][info][main] external/envoy/source/server/server.cc:392] exiting

I have not modified this file and I was under assumption that since it is in repo it should be good but apparently it is not, appreciate if you could review it and let me know what cab be done.

@@ -0,0 +1,112 @@
// Copyright 2018 Istio Authors
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this to /tests/util (top level)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do.

--execute=templates/configmap.yaml --values install/kubernetes/helm/istio/values.yaml > /tmp/istio-configmap.yaml
kubectl create -f /tmp/istio-configmap.yaml
helm template ${ISTIO_GO}/install/kubernetes/helm/istio --namespace=istio-system \
--execute=charts/ingress/templates/service.yaml --values install/kubernetes/helm/istio/values.yaml > /tmp/istio-ingress.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this /tmp dir usage looks a little problematic from security POV. I would take a look at man mktemp(1) page has example usage. I don't know the context in which this is used (is it in the gate, on a local dev environment, on a shared machine?)

Depending on the answer it may not matter. It should be relatively easy to make a secure file in tmp with mktemp

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will use istio's LOG_DIR variable for the location of temp files.

@sdake
Copy link
Member

sdake commented Apr 9, 2018

@xiaolanz racetest failure, can you analyze. TX

Copy link
Contributor

@costinm costinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally very nice, will allow more local coverage !

@@ -7,10 +7,15 @@ set -euo pipefail
# Based on circleCI config - used to reproduce the environment and to improve local testing

# expect istio scripts to be under $GOPATH/src/istio.io/istio/bin/...
export TOP=$(cd $(dirname $0)/../../../..; pwd)

export TOP=${GOPATH}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${TOP:-${GOPATH}}

@@ -128,49 +133,82 @@ function startLocalApiserver() {
# make sure apiserver is actually alive
kill -0 $(cat $LOG_DIR/apiserver.pid)

echo "Started local etcd and apiserver !"
# Really need to make sure that API Server is up before proceed further
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can leave it the way it was - start, do other things (build docker images in case of CI), waitForApiserver, apply config maps.

But we can refine it - 3 seconds is not that bad.

kubectl get nodes 2>/dev/null || startLocalApiserver
}

function createIstioConfigmap() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, location should not be hardcoded - we can add some env variable to control it (PILOT_CONFIG ?).

We may also do (1) - and watch so we handle config changes, but (2) should be done either way, we also support
cases where there is no k8s.

Copy link
Contributor

@costinm costinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@istio-testing istio-testing added lgtm and removed lgtm labels Apr 11, 2018
@sbezverk
Copy link
Contributor Author

/hold cancel

@istio-testing istio-testing removed the do-not-merge/hold Block automatic merging of a PR. label Apr 13, 2018
@sbezverk
Copy link
Contributor Author

/test istio-unit-tests

@costinm
Copy link
Contributor

costinm commented Apr 13, 2018

can you merge master - lint and ome if the tests is failing

@@ -0,0 +1,23 @@
admin:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create a subdir - it may confuse the other 'file-based' adapter which reads from this directory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

return err
}

func TestClusterRegistry(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think util is the right directory for this test. Can you move it next to cluster registry ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved both to tests/local since both of these tests needs to run in local environment.

@@ -0,0 +1,247 @@
// Copyright 2018 Istio Authors
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same - either in the v2 ( next to xds_test), or some other place - but likely not tests/util

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@ldemailly
Copy link
Member

/lgtm

@istio-testing
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: costinm, ldemailly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sbezverk
Copy link
Contributor Author

@costinm I think it is ready, WDYT?

@istio-merge-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@istio-merge-robot
Copy link

Automatic merge from submit-queue.

@istio-merge-robot istio-merge-robot merged commit 60c35a4 into istio:master Apr 13, 2018
jwendell added a commit to jwendell/istio that referenced this pull request Apr 18, 2018
make init shouldn't actually build any binary.
That was introduced in istio#4773.
istio-merge-robot pushed a commit that referenced this pull request Apr 19, 2018
Automatic merge from submit-queue.

Don't build anything in the init phase

make init shouldn't actually build any binary.
That was introduced in #4773.
ozevren added a commit that referenced this pull request Apr 23, 2018
* Do not add jwt and authn filter for TCP listner type (they are http filters) (#5034)

* Improve virtual service validation (#5027)

Automatic merge from submit-queue.

Improve virtual service validation

Adds more tests to virtual service validation and fixes a few issues. This PR improves the coverage but more test cases need to be added.

* istioctl: fix default namespace handling for (de)register commands (#4493)

* istioctl: fix default namespace handling for (de)register commands

PersistentPreRun is only invoked for the leaf subcommand in the
command chain and parent PersistentPreRun are skipped. In case of
(de)register's PersistentPreRun, this skipped default namespace
handling. Fortunatly, (de)register was only calling getRealKubeConfig
which was already handled by the parent command.

* fix lint errors

* Removed duplicate core v1 import. (#4957)

Automatic merge from submit-queue.

Removed duplicate core v1 import.

* update the --grpc-host-identities=istio-ca to preserve the default behavior. (#4972)

Automatic merge from submit-queue.

update the --grpc-host-identities=istio-ca to preserve the default behavior.

We still need to set default value of --grpc-host-identities=istio-ca to support mesh expansion customer.

* Support multiple route rules versions in Bookinfo e2e tests (#4896)

* move route-rule-reviews-90-10 and route-rule-reviews-80-20 with all other routing rules

otherwise, complex logic will be required to test them on v1alpha3 and v1alpha2

* added v1alpha3 versions of route-rule-reviews-80-20/90-10

* add TestFlags to e2e test framework

* add reprocessRule

* split demo_test.go into demo_test.go and main_test.go

* refactor: extract getPreprocessedRulePath

* add TestFlags definition and initialization

* remove adding default rules from setUpDefaultRouting

* remove deleting default rules on testConfig cleanup

* remove adding default routing rule

* print the migration rate in TestVersionMigration

* add setting default rules in TestVersionMigration

* ignore an error in deleting a rule that was deleted before

ignore the error that will happen since the fifty rule redefines
"reviews-default" rule and is deleted

* add default rules to test version routing and test fault

* add handling of rules config versions

* refactor: extract allRules variable

* add reviewsDestinationRule for v1Alpha3

* Skip -> Skipf

* Remove forCA from CsrRequest. (#5018)

Automatic merge from submit-queue.

Remove ForCA from CsrRequest.

ForCA is not used by Citadel any more.

Instead, Citadel decides whether to sign certificate for workload or CA from the flag `sign-ca-certs`:
https://github.com/istio/istio/blob/master/security/cmd/istio_ca/main.go#L218

* Use pod IP instead of host IP (#4988)

Automatic merge from submit-queue.

Correct IP for pilot debug tool.

* Remove vendor files from .gitignore (#5048)

* update Go control plane (#5053)

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* updated istio/api repo version (#5050)

* updated istio/api repo version

* code review comments

* Add myself to install/OWNERS (#5046)

Automatic merge from submit-queue.

Add myself to install/OWNERS

* cleanup unused files (#5042)

Automatic merge from submit-queue.

cleanup unused files

istio-pilot-e2e-v1alpha3 is not run anymore. istio-pilot-e2e-envoyv2-v1alpha3 replaced that test

* Don't build anything in the init phase (#5035)

Automatic merge from submit-queue.

Don't build anything in the init phase

make init shouldn't actually build any binary.
That was introduced in #4773.

* [Part2] Added `ImagePullSecrets` to `SidecarInjectionSpec` (#5002)

Automatic merge from submit-queue.

[Part2] Added `ImagePullSecrets` to `SidecarInjectionSpec`

This PR updated the test cases related to SidecarInjectionSpec.

Fixed #4870

/cc @ayj @yusuoh @linsun

* Mixer filter fixes (#5012)

* Add proxy instances to plugin input params

* Fix yet another slice handling issue

* Update mixer cluster names

* Fix mixer cluster addresses

* Fix mixer cluster addresses #2

* Put registry and secrets into a nodeagent package. (#5054)

Automatic merge from submit-queue.

Put registry and secrets into a nodeagent package.

Two sub-package under nodeagent/
- `registry/`, is to handle the workload pod creation, deletion, etc, contains interaction with flexvolumedriver.
- `secrets/`, is to handle the secrets management, containing envoy SDS API interaction.

* Fix pilot gRPC port in mTLS mode (#4998)

* Use 15011 for pilot gRPC port when mTLS enabled.

* Update bootstrap test.

* Add tls context to bootstrap v1 (ingress using this).

* Update bootstrap golden data.

* Add a prow test for bookinfo v1alpha3 route rules (#5049)

Automatic merge from submit-queue.

Add a prow test for bookinfo v1alpha3 route rules

* Sets up Pilot and remote cluster for multicluster (#4997)

Automatic merge from submit-queue.

Sets up Pilot and remote cluster for multicluster

This change creates a secret and configmap in order
to start pilot in mulitcluster mode.  It also creates
the necessary resources on the remote cluster to deploy
applications there.

* Update helm chart version to 0.8.0 (#5030)

* simple e2e test: fix error output and exit condition (#5060)

* [pilot] Some minor cleanup of bootstrap code (#5071)

Automatic merge from submit-queue.

[pilot] Some minor cleanup of bootstrap code

* Start GRPC server using native go stack, cleanup auth (#4867)

* Refactoring the init code for grpc.

* Merge cleanup

* Add the secure grpc port (also serving https)

* Revert cleanup

* Refactor and fix AZ retrieval, add test

* Change default since the tls-via-envoy uses that, remove verbose log

* Allow custom DNS names for pilot

* Panic if certs not found an policy requires it

* Setting the add to emtpty will disable the mtls listener

* Revert vendor again

* Fix test failure

* Typo

* Make sure previous logs are dumped, it seems pilot may crash

* Manual fix of fmt -c

* Better fix for 'prev log'

* Lint errors

* routes no longer working

* Mount certs to pilot pod

* Lint fix and revert pod logs, it's used for something else

* Format

* Add missing transitive dependency envoyproxy/go-control-plane (#5076)

Automatic merge from submit-queue.

Add missing transitive dependency envoyproxy/go-control-plane

Not entirely sure how these were missed previously. Regardless,
running `dep ensure` adds them to vendor (no changes to our lock
file since they're transitive).

* Remove EUC validation code (#5080)

Automatic merge from submit-queue.

Remove EUC validation code

Issue: #4744

* renamed e2e-bookInfoTests-v1alpha3.sh -> e2e-bookInfoTests-envoyv2-v1alpha3.sh (#5087)

Automatic merge from submit-queue.

rename e2e-bookInfoTests-v1alpha3.sh -> e2e-bookInfoTests-envoyv2-v1alpha3.sh

* Revert "[pilot] Some minor cleanup of bootstrap code (#5071)" (#5090)

This reverts commit d5fa2b7.

Conflicts:
	pilot/pkg/bootstrap/server.go

* Enable RBAC (#5082)

* Use zipkin 2.6.0 in istio. (#4726)

* Modernize istio-remote helm chart (#5083)

Automatic merge from submit-queue.

Modernize istio-remote helm chart

Modifies endpoints and services to match master
This PR further changes the Makefile target to use istio-${service}.istio-system
This PR further adds in the CA service and deploys it by default

* Remove gcloud docker calls (#5092)

Automatic merge from submit-queue.

Remove gcloud docker calls

Fixes #4797

* Add per proxy pilot querying (#5096)

* Add per proxy pilot querying
This is the final chunk of functionality for the base proxy-config command. It adds the ability to query Pilot for a specific proxy as well as the full mesh support.

* Linting...

* Correct the reference file path in galley (#5000)

* Add secret controller for multicluster (#5017)

* Vendor changes adding Informers and Listers

* Secret Controller code

* Linter detected issues

* Vendor update related sha change

* Adding final bits to the controller

* Fixing controller startup code

* Adding required RBAC rules to watch for secrets

* Refactor Cluster Store initialization place

* Fixing Unit test failure

* Fixing Unit test failure

* Addressing comments part #1

* Fixing Unit test

* Switching to different type of Informer

* Add create k8s_cr.Cluster object

* Fixing if statement

* Fixing lint error

* Cosmetic changes

* Fixing lint error

* Tests reproing 503s (was 404s) during routerule apply - with t.Skip() until we have fix (#1041)

Reproduces 3 classes of bugs. with t.Skip() for now/until they are fixed.

* Multicluster deployment of Apps on remote (#5099)

Automatic merge from submit-queue.

Multicluster deployment of Apps on remote

This change deploys the remote applications in a multicluster
test.   It alps fixes a couple issues with the remote.yaml
and a typo from a prior PR. It makes some fixes to match up
with PR5083

* Use iptables TPROXY instead of REDIRECT for inbound traffic (#4654)

* Pilot: Support running Envoy with CAP_NET_ADMIN to support TPROXY

Add iptables as a dependency to the istio.deb package.

Signed-off-by: Romain Lenglet <romain@covalent.io>

* Pilot: Support iptables TPROXY instead of REDIRECT for inbound traffic

Add iproute2 as dependency to the istio.deb package and the proxy_init
Docker image.
Add a "-m" command-line flag to istio-ipstables.sh to select the inbound
traffic interception mode ("REDIRECT" or "TPROXY").
Fix the usage doc for the other command-line options.

Signed-off-by: Romain Lenglet <romain@covalent.io>

* Fix deb/* Makefile targets

Signed-off-by: Romain Lenglet <romain@covalent.io>

* Pilot: Configure transparent proxy redirection from proxy config

Configure the redirection mode (redirect vs. tproxy) in ProxyConfig.
Pass the redirection mode to Pilot in the Node's metadata, in environment
variable ISTIO_META_INTERCEPTION_MODE.

Fix Envoy bootstrap to remove the ISTIO_META_ prefix from metadata
key names, instead of ISTIO_META.

Signed-off-by: Romain Lenglet <romain@covalent.io>

* Pilot: Parse Node Metadata and store it in Proxy

Signed-off-by: Romain Lenglet <romain@covalent.io>

* Update pilot/pkg/kube/inject tests

Fix pilot/pkg/kube/inject tests to use the new templates.
Add unit tests for the ProxyConfig.InterceptionMode.

Signed-off-by: Romain Lenglet <romain@covalent.io>

* Pilot: Define sidecar.istio.io/interceptionMode annotation

Define sidecar.istio.io/interceptionMode to override the
ProxyConfig.InterceptionMode mesh-wide setting.

Signed-off-by: Romain Lenglet <romain@covalent.io>

* added missing external service definition (#5094)

* Fix the node agent E2E test, manually tested. (#5063)

Automatic merge from submit-queue.

Fix the node agent E2E test, manually tested.

Fixed several things to get it work:
- Update outdated flags value.
- In `start_app.sh`, retain the node agent process log, instead just run it in background.
- Make sure the initial root and cert is loaded.

How to test this:
- `make docker && make docker.push`
- `go test -v  istio.io/istio/security/tests/integration/nodeAgentTest  --tag $(git log -1 --format="%H")  --hub gcr.io/<your-project> -kube-config ~/.kube/config --skip_cleanup`

We should also figure out how to not run `apt-get` install for every test run. Will follow up with key cert generation issue and then ensure it's stability before re-enabling it.

cc @wattli

* Bug fix in stackdriver adapter. (#5026)

Automatic merge from submit-queue.

stackdriver adapter bug fixes and clean up

Several bug fixes and cleanups in stackdriver adapter:

- add duration type into distribution processing, which is used for latency metrics.
- remove a duplicated layer of loop.
- remove metric kind override since now adapter could deal with not only custom metrics.
- remove a log line which overwhelms mixer logs.

* Only query the specific trace using the provided x-client-trace-id tag (#5066)

Automatic merge from submit-queue.

Only query the specific trace using the provided x-client-trace-id tag

This change should make the trace query more efficient as it should only retrieve the single required trace instance.

cc @kyessenov @nmittler 

Signed-off-by: Gary Brown <gary@brownuk.com>

* integrate circle ci with testgrid (#4649)

Automatic merge from submit-queue.

Integrate circle CI with Testgrid

The integration will start with mixer and simple e2e tests. Will expand to other jobs in subsequent PRs.

Results are now available on
https://k8s-testgrid.appspot.com/istio#circleci-e2e-mixer
https://k8s-testgrid.appspot.com/istio#circleci-e2e-simple

Those are test results from this PR exclusively. After this is merged, the results will be more interesting.

The `ci_to_gubernator` binary is built from istio/test-infra#769. Feel free to also take a look.

* Fixing postsubmit (#5104)

* Implement Citadel prometheus monitoring feature. (#5015)

Automatic merge from submit-queue.

Implement Citadel prometheus monitoring feature.

This monitoring feature exposes a service (port 9093) about Citadel status to prometheus.
Yaml files are changed in #5072.
Tests will be added shortly.

* let stackdriver adapter figure out project metadata by itself (#5057)

Automatic merge from submit-queue.

let stackdriver adapter figure out project metadata by itself

This will make stackdriver adapter config generalized and usable for different projects.

* Fix labelNames getting filled in stackdriver logging. (#5058)

Automatic merge from submit-queue.

Fix labelNames getting filled in stackdriver logging.

Added LabelNames in testdata/stackdriver.yaml file
Also, fixed that if label was not string, it was getting filled as null.

* Don't attempt to qualify the wildcard domain. (#5106)

* Don't attempt to qualify the wildcard domain.

* fmt

* pilot-agent: add a flag to disable internal telemetry (#5101)

Automatic merge from submit-queue.

pilot-agent: add a flag to disable internal telemetry

Enabled by default, but allows to turn off telemetry.

Signed-off-by: Kuat Yessenov <kuat@google.com>

* Add tcp proxy support for multi-cluster environment (#4694)

Automatic merge from submit-queue.

tcp proxy support in a multi-cluster environment.

Services need to maintain per cluster service IPs.  For sidecars running in a cluster, cluster-specific service IPs should be used to build destination_ip_list for envoy listeners.

* Add missing namespace to citadel template (#5108)

Automatic merge from submit-queue.

Add missing namespace to citadel template

* prune old version resources that no longer exist (#5107)

Automatic merge from submit-queue.

prune old version resources that no longer exist

* [vendor-change] CloudWatch Mixer adapter (#4617)

Automatic merge from submit-queue.

[vendor-change] CloudWatch Mixer adapter

Adding an adapter to send metrics to cloudwatch

* Enable Ingress/Egress gateways in Helm for bookinfo demos (#5120)

Automatic merge from submit-queue.

Enable Ingress/Egress gateways in Helm for bookinfo demos

* Consume labeled multicluster secrets on startup (#5117)

Automatic merge from submit-queue.

Consume labeled multicluster secrets on startup

This patch when run against istio.yaml or istio-auth.yaml
runs in the new config mode using only labels rather than
configmaps.  The configmap functionality can be removed in
0.9.

* Add a linter check to make sure types.go are generated. (#5110)

Automatic merge from submit-queue.

Add a linter check to make sure types.go are generated.

addresses #4418

* Remove outdated manifests from install/kubernetes (#4882)

* Remove orig_ manifests

* Remove istio-mixer-validator and istio-mixer-with-health-check manifests

* Remove unwanted manifests before archiving

* Remove istio-sidecar-injector.yaml from install/README.md

* Remove *one-namespace*.yaml from install/README.md

* Make helm-generated manifests overwrite updateVersion_orig.sh manifests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants