* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* [test-framework] Design sketch for test framework.

* [test-framework] Reorgnanization and linter fixes (#5128)

* [test-framework] Adding pilot test showcase (doesn't build)

* [test-framework] Adding mixer test showcase (doesn't build)

* [test-framework] Get the code to build.

* [test-framework] Cleanup the dependency model. (#5212)

* Cleanup the dependency model.

+ Colocate all (current) dependencies.
+ Tighten the Dependency interface.
+ Introduce internal.Stateful to reduce the surface area of Dependency.

* Cleanup and linter fixes.

* [test-framework] Adding basic cluster app (#5257)

* [test-framework] Adding cluster App creation (#5342)

* [test-framework] Add example to simple showcase test (#5369)

* add new changes to simple showcase test

* Fix typo

* [test-framework] Rationalize FortioApp model. (#1) (#5452)

* Rationalize FortioApp model.

- GetFortioApp returns a single instance of DeployedFortioApp and error.
It uses a name pick the app directly, similar to the GetApp model.
- Add *OrFail overload, similar to GetApp case.
- Add GetFortioApps which has a label selector, and returns an array of
deployed apps.

* More cleanup of the Fortio code and get the code to compile.

* [test-framework] First pass cleanup of the model (#2) (#5451)

* First pass cleanup of the model:

- Add CallOrFail to DeployedApp for expedient test authoring.
- Replace the Mixer mock adapter with a mock PolicyBackend. This is much
easier to manage from a testing standpoint.
- Remove label based custom logic. Labels are used for filtering only.
- Align Mixer showcase test with the Pilot app model. This way, we can
author Mixer integrations tests by applying configuration and controlling
app/Mixer backend behavior.

* Accommodate CR feedback.

* [test-framework] Remove charts/DeployedIstioComponent and Environment cleanup (#5474)

* Remove charts and DeployedIstioComponent.

* Cleanup the Environment interface and enforce methods in local and
cluster implementations.

* [test-framework] Refactor the driver interface and implementation. (#5475)

* [test-framework] Cleaning up App API to simplify URL construction (#5487)

* [test-framework] API and command-line cleanup. (#5510)

* Improve the command-line handling and remove ApiServer & Helm code.

* - Check double-run in driver.Interface.Run().
- Add support for suite level dependency processing.
- Use scoped logs in the driver.

* [test-framework] Add internal interface for the environment (#5523)

* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* [test-framework] Adding bootstrap for local Envoy and Pilot App (#5513)

* [test-framework] Add health check to local envoy agent at startup (#5551)

Also fixing a bug in the envoy yaml template that was preventing the
configuration of multiple ports to a single instance.  Expanded the
test to cover this.

* [test-framework] Switch from testing.T => testing.TB and uniformize *OrFail. (#5716)

* [test-framework] Further rationalize driver model. (#5742)

* Further rationalize driver model.

- Extract out a "Context" to be used internally within the testing code. This is mainly useful to break cyclic dependency between the environment and the driver code.
- Cleanup the logging code: Most of the code here is not needed anymore: as the standard set of logging flags already take care of writing logs to well known locations
- Similarly, tmp code can be cleaned up now, to mainly focus on per-component state storage. Moved the code within driver folder to reduce fragmentation.

* Move the GetEnvironment method from the driver interface to Context.

The move allows us to use context object as the single handle to pass
through the rest of the test-framework stack.

* Minor naming fix.

* [test-framework] Revamp the internal machinery of the test framework. (#5804)

- Simplify the dependency model. Instead of dependencies having logic, they simply are enums.
- Rename GetEnvironment to AcquireEnvironment and change semantics to at most one per test. With this model, we can reset the internal state of dependencies during the acquisition of the environment.
- Implement a loose object model that opt-in to test framework services through interface implementation. The three services that can be opt-in to are: receiving user supplied configuration, cleanup, and reset with every environment acquisition.
- Implement a resource tracker for supplying these services.
- Switch to a common, internal environment interface, as opposed to environment specific internal interfaces. This switches the dispatch model so that environments are inherently aware of how each dependency work, rather than dependencies adjusting themselves for each environment. In effect, the internal structure of the code should be simplified.
- Use os.TempDir() as the default workdir.
- Fix formatting of help output.
- Switch internal.TestContext interface to be a struct to reduce code clutter.

* [test-framework] Implement Mixer related infrastructure for local environment. (#5805)

* Add local-Mixer infrastructure and implement a basic test to cover.

* Accommodate CR feedback.

* Separate out the settings.

* [test-framework] Adding local pilot (#6077)

* [test-framework] Add cluster support to the test framework. (#6179)

* Add cluster support to the test framework.

* CR feedback.

* Minor linter fixes.

* [test-framework] Integrating local Pilot and Envoy (#6332)

- Updated configuration of Envoy to use discovery v2 API on Pilot

- Adding a discovery proxy to allow interception of responses from
Pilot. This will provide the ability to modify the Envoy configs to
support local testing.

- Updated the agent test to incorporate a local pilot. Also starting
to hash out how to determine whether or not an Envoy has received
a configuration update.

* [test-framework] Refactoring pilot test application (#6489)

Simplifying the structure to make the code more readable and to make it
better fit in with the application model of the new local agent.

* [test-framework] A few fixes for local envoy code (#6537)

* [test-framework] More robust port reservation (#6736)

This PR reserves a block of ports, so that future attempts to pick
a port will not collide with previous attempts.

* [test-framework] Add control RPC to test service (#6737)

The RPC allows us to simulate in-mesh traffic from one service to
another.

Creating a copy of the pilot test service under pkg/test. This is a
more appropriate home for the code and also avoids changing the rest
of the code that depends on the old service.

* [test-framework] Support local in-mesh traffic (#6743)

There are a lot of changes here that are interrelated.  Here's the summary.

1) Adds concept of a protocol client, which is used to intercept outbound
URLs. The interception is performed by the proxy which modifies the
URL host:port so that an outbound call from service A to B will first go
through service A's Envoy proxy.

2) Moved the discovery response filtering logic into the application proxy,
since the additional logic requires state from the proxy.

3) Updated the discovery filtering logic to intercept the outbound listeners
created by Pilot to enable communication with other services
(e.g. so A can talk to B). Pilot uses virtual listeners which are not
actually bound to a port (since outbound traffic would typically be
redirected to 15001 in a cluster configuration). The updated logic
modifies these listeners by assigning a newly reserved port and forcing
the listener to bind to that port. The new port is then stored in a
port map, which is used to modify the URLs on outbound traffic from the
application.

* [test-framework] Remove tagging/label support. (#6975)

* Remove tagging/label support.

* remove ignore

* [test-framework] Add API Server support. (#7276)

* [test-framework] APIServer + Galley tests (#7277)

* [test-framework] Code cleanup: Move and simplify the core driver code. (#7286)

* Refactor the core driver code:

+ Created test/framework/ as the namespace of the internal/implementation
of the test framework.
+ Moved the top-level driver code to test/framework/driver
+ Simplified the code at the pkg/test level, leaving only framework.go
for top-level test framework operations.
+ Simplified code in framework.go to simply redirect to the driver.

* More code move:

+ Created framework/environments and moveed cluster & local there.
+ Renamed cluster to kubernetes.
+ Moved test/internal to test/framework/internal.
+ Moved test/tmpl to test/framework/tml.

Also renamed environment.Interface to framework.Environment.

* More shuffling:

+ Moved test/dependency =? test/framework/dependency.
+ Moved framework/environment.go => framework/environment/environment.go
This is needed to break the dependency cycle.
+ Moved environment variable flags, args code to framework/settings.
+ Moved test/framework(formerly operations.go) to test/api.go.
+ Moved driver code from framework/driver/... to framework/

* Fix the mutex error.

* Rename construct => new

* [test-framework] Support non-HTTP ports in local agent (#7419)

Also doing some other cleanup and build fixes.

* [test-framework] Componentize the test framework. (#7344)

* Refactor and componentize the test-framework.

* Accommodate CR feedback.

* [test-framework] Move pilot to components. (#7515)

* [test-framework] Some minor cleanup for pilot component (#7536)

* [test-framework] Adding support for component dependencies (#7598)

- Added Component interface, which is now implemented by all components

- Separate component registries for local and kubernetes

- Updated the dependency Tracker to initialize all dependencies of
a component before initializing the component, itself.

* [test-framework] Integrating test apps into framework (#7628)

* [test-framework] Updating jsonpb dependency

* [test-framework] fixing linter errors in generated files

Needed to rename the files so they'd be ignored.

* [test-framework] Removing t.Skip calls in unit tests.

* [test-framework] removing accidental edit

* part 2 TCP routing cleanups/bug fixes for header matcher (#7236)

* update envoy sha - fix memory leak

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Fixing assorted bugs in service entries using filter chain matches

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* format

* bug fix

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* lint

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* lint

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* multicluster bug fixes

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* backward compatibility

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* use new header match specifier

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* mixer: add sidecar health probe (#7102)

* sidecar health

Signed-off-by: Kuat Yessenov <kuat@google.com>

* typos

Signed-off-by: Kuat Yessenov <kuat@google.com>

* remove tracing of health checks

Signed-off-by: Kuat Yessenov <kuat@google.com>

* change to 15093

Signed-off-by: Kuat Yessenov <kuat@google.com>

* Add proxy version to the proxy-status command (#7269)

* use new websocket option and remove deprecated one (#7247)

* use new websocket option and remove deprecated one

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* backward compatibility

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* consistency

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* cleanup

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* dont append upgrade configs

* remove requestedServerName attribute (#7278)

it does not work, requires debugging and possibly implementation, will not be in time before the release

* Handle virtual service sni_hosts matches in gateway. (#7192)

* dirty poc

* working poc

* comments + cleanup

* lint

* add simple e2e test to egress

* minor refactoring

* cleanups, lots of comments

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* more duplication for clarity

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* simplifying gateway opaque tcp logic

* final cleanups

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* nil pointer check

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Add add operation into expression language (#6776)

* add op.

* fix duplicated test.

* Mixer Conditional Quota (#7265)

* add tests for conditional and unknown quota

* add conditional and unknown quota

* Option to disable pilot sidecar (#7280)

* Remove whitespace between host: and port in 'istioctl authn tls-check' (#7084)

* Remove whitespace between host: and port

* Typo

* enable upgrade config only for 1.0 proxy (#7300)

* enable upgrade config only for 1.0 proxy

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* lint

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Add debounce to push (#7253)

* Fix log, add debounce

* If env is 0, disable debounce

* Add missing config file

* Safer code, failsafe

* Update comments and remove unused metrics

* Lower verbosity

* Increase quiet interval - k8s throttling seems to kick in, 100 ms too little

* Verbose message triggered for tcp services

* Format and adjust timer for debounce, a second was too long.

* Fix backward compat to 0.8

* push to sidecars when detecting jwt pubkey change (#7303)

* Added `priorityClassName` to Istio chart. (#7258)

Kubernetes >=v1.11.0 will create two PriorityClass, including
system-cluster-critical and system-node-critical, it is better
to configure this in order to make sure your Istio pods will
not be killed because of low prioroty class.

Refer to https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
for more detail.

* make install istio release version configurable (#7268)

* update galley to self-register and reconcile the validatingwebhookconfiguration (#7234)

* update galley to self-register and reconcile the validatingwebhookconfiguration

* bin/fmt.sh

* comment exported functions

* fix bug

* fix spelling in error message

* fix spelling error and rbac rules

* add explicit deletion of webhook configuration

* refactor code and improve cleanup handling

* update tmp change

* fix lint and format errors; clean-up dead code

* improve logging and fix config reload bug

* improve tests

* remove unused var

* remove explicit deletion of webhook configuration.

* add more comments and rename some functions

* Remove the unnecessary 'creationTimestamp' fields in sample's yamls (#7210)

* Fix istio-remote sidecar-injector-configmap rendered yaml parse error (#7260)

* Fix istio-remote sidecar-injector-configmap rendered yaml parse error

* Fix istio chart sidecar-injector-configmap rendered yaml parse error

* Change tracing chart to use tracing.provider to select backend (#7120)

* Do not allow short names in gateway hosts (#7301)

* Do not allow short names in gateway hosts

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* allow *

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* lint

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Change instances of `password` to `passphrase`

* Stress test updates (#7306)

* Current changes to the stress test environment

* Remove old files

* Add keepalive (#7332)

* Add keepalive

* Update golden

* Generate LICENSES, make branch required (#7289)

* add rbacconfig validation (fixing istio-pilot crash) (#7324)

* Remove the clusterregistry configmap usage in favor of labelled secret (#6454)

- this aligns with current docs & preferred remote cluster config
  reference.

* Check RbacConfig for nil target field in authz plugin.

* Update helm to 2.9.1 and open-config-agent/opa to 0.8.2 (#7335)

* Logic and race test fixed

* Mock JWT and JWKS data for end-user authentication demo/testing (#7347)

* Add sample JWT and JWKS data for simple end-user authentication demo

* Clean up readme

* Correct example

* Correct script comments

* remove unused v1alpha3 rules

* add rule specific for consul (#7365)

* Add HPA config for istio-policy and istio-telemetry (#7218)

* WIP: Initial HPA config for Mixer

* Minor cleanup

* Update Mixer test to pull logs from potentially multiple mixers

* lint fix

* Remove memory from autoscaling resource consideration

* Limit Prometheus discovery to relevant namespaces (#3866)

This change limits the Prometheus discovery to the relevant namespace(s)
instead of dropping targets during relabeling. The gain is less load on
the Kubernetes API.

* Added annotaion of `scheduler.alpha.kubernetes.io/critical-pod: ""`. (#7312)

This can make sure the pod will not be preempted after it was deployed
on one node.

* Addressing flakes in listener Conflict Test (#7336)

* Addressing flakes in listener Conflict Test

- Adding a sleep between service pushes to make sure they get different
creationTimestamps.

- Use the push_status endpoint rather than scraping pilot logs.

* support multiple pilots and bump up retries.

* Update_Dependencies (#7359)

*  Ask for cluster state archive in bug report template (#7281)

* Ask for cluster state archive in bug report template

* Add preface

* Set to istio.io

* Do not set ingressService if ingress is disabled (#6100) (#7353)

* make ansible playbook run in local istio version (#7354)

* Run dep ensure (#7370)

* update istio api (#7372)

* Convert prometheus and grafana ingress to ingressgateway (#7346)

* Convert prometheus and grafana ingress to ingressgateway

* Merge telemetry gateways into single spec

* Address typo

* Mock JWT and JWKS data for end-user authentication demo/testing (#7347) (#7374)

* Add sample JWT and JWKS data for simple end-user authentication demo

* Clean up readme

* Correct example

* Correct script comments

* Ingress cleanups (#7222)

* Ingress and gateway updates

* Add gateways

* Remove the old ingress chart

* Don't configure https port automatically

* Remove for now the istio services ingress, roll back the old ingress.

* Really add back the ingress chart

* Add preinstall annotations and move CRDs (CRDs are not specific to pilot, but istio general)

* Add missing files (autoscaler for pilot as well, seems I missed it)

* Attempt to fix failing auth server

* use http2

* Move back crds to pilot, helm doesn't like them in templates

* use crd-install hook, up helm, more logs to debug failing test

* helm version is in 2 places

* Bump helm to 2.9.1

* Use pre-install

* Let's try CRDs without dry-run

* Helm crd creation fixes

* Autoscale doesn't work well with the default 0.1 alloc - causing test failures

* Fix failing test

* Broken build with attempt to support testing multiple helm versions

* More attempts to get CRDs working in all cases

- revert using pre-init for security - it doesn't help and it's not
cleaned on delete. System eventually recovers.

- switch test to helm 2.9 - we should test with released version
Do the crd install ahead of time.

- added explicit control for CRD inclusion, so helm template with
pre-2.10 works as before. When using kubectl apply it is safe to have
the crds included (but still better to apply them before install)

- update dep to avoid lint failure

* Prow uses different base dir

* Bad conflict resolution

* Fix remaining missing licenses (#7382)

* Add validation metrics and pilot monitoring metrics to dashboards (#7291)

* Add validation metrics and pilot monitoring metrics to dashboards

* Update e2e_dashboard

* Filter galley error queries

* improves sort performance (#7399)

* convert-networking-config for Ingress (#7366)

* convert-networking-config for Ingress

* Lint

* Typo

* Usage instructions and command re-named

* Generate only VirtualService

* Allow SNIs in Istio MTLS (#7396)

* Allow SNIs in Istio MTLS

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* more changes

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* tests

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* remove redundant whitespace (#7385)

* Remove post-install Mixer CR job (#7400)

* Remove post-install CR job

* maintain old name for accesslog for upgradability

* Normalize shebangs in shell scripts (#7329)

* "#!/usr/bin/env bash" -> "#!/bin/bash"

* "#! /bin/bash" -> "#!/bin/bash"

* "#! /bin/sh" -> "#!/bin/sh"

* Skip TestRedisQuota test due to flakiness (#7406)

* Update servicegraph reporter query label (#7421)

* Fix experimental metrics cli (#7403)

* Apply Istio mTLS to subset-level port-specific traffic policies (#7423)

* Allow SNIs in Istio MTLS

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* more changes

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* tests

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* apply Istio mTLS to subset port level settings

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Merge collab-galley to master. (#7371)

*  Separate generated metadata into their own, top-level package: pkg/metadata

- Introduce builder pattern to the core schema types that are used by pkg/metadata,
  to ensure runtime immutability after build.
- Move the Kubernetes related machinery to their own package, pkg/kube/source, to
  separate it from core Kubernetes related abstractions and utility code.
- Bolster tests and cleanup tests accordingly.

* Add builder support to InMemory snapshot. (#3)

* Add builder support to InMemory snapshot.

* Clone resurces as well.

* Use proto.Clone, instead of manual copying.

* Add a testing server for MCP. (#4)

* Add a testing server for MCP.

* Add overridable port field.

* Don't parse the port number.

* Minor bug & linter fix (#6)

* Fix a bug in the in-memory source implementation where the item itself
was missing when publishing already-existing items during the initial
sync phase.

* Fix lint issue.

* Updates to acommodate merge.

* Update Gokg.lock.

* A few improvements to ControlZ (#7424)

* A few improvements to ControlZ

- Add the RegisterTopic function which makes it possible to have packages register themselves into
ControlZ in a manner similar to how we support registering logging scopes.

- Add the Stop function which makes it possible to kill ControlZ wtihin a process. This is mainly useful
for test scenarios.

* Address CR feedback.

* fix panic on initial validation configuration load error (#7435)

The validation webhook panics during the error reporting path if the
initial webhook configuration or ca file is invalid (see stack trace
below).

This commit makes the error reporting more robust in case of
misconfiguration. The root cause for why configuration is invalid
still unknown.

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x121e091]
    goroutine 1 [running]:
    istio.io/istio/galley/pkg/crd/validation.(*Webhook).rebuildWebhookConfig(0xc4203a3560, 0x2c, 0x0)
        /workspace/istio-master/go/src/istio.io/istio/galley/pkg/crd/validation/config.go:89 +0x271
    istio.io/istio/galley/pkg/crd/validation.NewWebhook(0x1cf7b00, 0xc420401fa0, 0x28e19a0, 0xe, 0xe, 0x0, 0x0, 0x1bb, 0x7ffc9d9f0000, 0x1f, ...)
        /workspace/istio-master/go/src/istio.io/istio/galley/pkg/crd/validation/webhook.go:205 +0xabf
    istio.io/istio/galley/cmd/galley/cmd.validatorCmd.func1(0xc4202a0f00, 0xc4203fc9b0, 0x0, 0x5)
        /workspace/istio-master/go/src/istio.io/istio/galley/cmd/galley/cmd/validator.go:138 +0x498
    istio.io/istio/vendor/github.com/spf13/cobra.(*Command).execute(0xc4202a0f00, 0xc4203fc910, 0x5, 0x5, 0xc4202a0f00, 0xc4203fc910)
        /workspace/istio-master/go/src/istio.io/istio/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
    istio.io/istio/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4202a0a00, 0x0, 0x1b540a6, 0x13)
        /workspace/istio-master/go/src/istio.io/istio/vendor/github.com/spf13/cobra/command.go:846 +0x30a
    istio.io/istio/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4202a0a00, 0x6, 0x6)
        /workspace/istio-master/go/src/istio.io/istio/vendor/github.com/spf13/cobra/command.go:794 +0x2b
    main.main()
        /workspace/istio-master/go/src/istio.io/istio/galley/cmd/galley/main.go:27 +0x8x

* Fix error when calling lua script for fixed window in redisquota adapter (#7316)

* Fix error when calling lua script for fixed window in redisquota adapter
Add an e2e test for fixed window rule too

Make calculations in RedisQuota Test more accurate by using source numbers as compared to getting both source and destination and then dividing by 2.

(I saw a little discrepancy in source and destination numbers reported..
https://docs.google.com/spreadsheets/d/1bdP8QJoZuBEosIqfTd1BLG0M1tSQuaG3c5zU2bTIgqA/edit?usp=sharing, hence the change)

Made numbers for redisquota test little more lenient

Made numbers for redisquota test little more lenient

Skipping redis quota e2e tests for now :(

* Fixing lint error

* remove outdated comments (#6902)

* Adding Ctrlz support for MCP clients. (#7432)

* Adding Ctrlz support for MCP clients.

* Add no-lint to fix linter error.

* Add support for registering MCP clients to ConfigZ. (#7451)

* Remove outdated NOTES.txt files for prometheues and grafana (#7452)

* fix default --webhook-config-file value when validation is disabled (#7438)

* update api sha (#7454)

* Use stable version for prometheus. (#7444)

* authz: change the user field to use source.principal in principal. (#7440)

* authz: change the user field to use source.principal in principal.

* update sample yaml.

* address comments.

* Update Proxy SHA to latest with RDS fix. (#7457)

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

* Allow users to easily run istio ingress gateway in given namespace. (#7408)

* WIP adding istio ingress gateway chart

* WIP - to be tested but seems able to generate the gateway yamls

* put the condition in comments to allow kubectl apply

 - thanks Costin

* additional tweaks

Since default is LB service type, uncomment nodeport as it will create port conflicts when install a second ingress gateway.  Also need to refer pilot in its namespace.

* make validation reconciliation more resilient when (#7466)

Some of the post-install jobs were recently removed. This exposed a
latent race where re-installing istio.yaml caused temporary
unavailable of validation and injection services. Explicitly set the
deployment rollout strategy's maxUnavailable to zero to avoid no
webhook endpoints during rolling updates.

* Reduced 'small pilot' cpu and increased memory to compensate. (#7460)

Fixes #7459

* Address comments from #7408 #7469 (#7472)

* WIP adding istio ingress gateway chart

* WIP - to be tested but seems able to generate the gateway yamls

* put the condition in comments to allow kubectl apply

 - thanks Costin

* additional tweaks

Since default is LB service type, uncomment nodeport as it will create port conflicts when install a second ingress gateway.  Also need to refer pilot in its namespace.

* Address comments from 7408 #7469

* Load testing with 1000 ServiceEntries and 512 pods (#7439)

* Lock the push to avoid overalps, throttle

* fmt

* Add the env variables, fix build

* Pod/VM load testing fixes

* Add the new ready function, endpointz is very expensive

* Add quotes to the env

* fix build

* Spelling errors for lint

* Rename variable

* Fix the lock

* Merge from the test branch

* Fix 300s timeout for grpc streams (#7428)

* Fix 300s timeout for grpc streams

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* undo go cp bump

* more fixes

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* remove new files

* more undo

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* final undo

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Tests keep failing with timeout so no logs

* Revert the timeout fix to see if it hels the test

* Revert the 0 timeout as well

* build error

* Try again max grpc - leave idle timeout removed

* Bring back the IdleTimeout

* Remove IdleTimeout

* Make format

* update go control plane

* use stream_idle_timeout

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* undo gRPC version bump

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* build fix

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Add new example ratelimit yaml to be used from istio.github.io (#7485)

* Handle TLS as opaque TCP (#7394)

* only create filterchain for opaque tcp when there are no tls matches.

* don't generate config with duplicate runtime matches

* update test for change in how we handle https/tls (make implementation agree with api specs)

* Properly merge coverage files (#7430)

* Properly merge coverage files

* Re-enable code coverage. It was silent

* Skip hypersistio as it is failing

* Update codecov requirement

* update api sha, proxy sha (#7509)

* prepare for the new kiali that will support istio 1.0 (#7510)

* Several fixes for Kiali integration (#7498)

- Use a non-namespaced PROMETHEUS_SERVICE_URL (Kiali hardcodes istio-system kiali/kiali#350)
- Enable URL configuration for Grafana and Jaeger (Kiali tries to autodetect a service with an ExternalIP)
- Use a GRAFANA_DASHBOARD that really exists

* Revert "mixer: add sidecar health probe (#7102)" (#7507)

This reverts commit 86dc1496ef02591ce2f6b2490b161841733e1ffc.

* Move jaeger specific values under the tracing.jaeger value node (#7484)

* Fix deployment namespace #7446 (#7477)

Signed-off-by: Sébastien Prud'homme <sebastien.prudhomme@gmail.com>

* delete duplicate app label (#7513)

Signed-off-by: Kuat Yessenov <kuat@google.com>

* istioctl command prompt world spell wrong (#7471)

* Address pilot namespace comment when running in non istio-system ns #7470 (#7508)

* WIP adding istio ingress gateway chart

* WIP - to be tested but seems able to generate the gateway yamls

* put the condition in comments to allow kubectl apply

 - thanks Costin

* additional tweaks

Since default is LB service type, uncomment nodeport as it will create port conflicts when install a second ingress gateway.  Also need to refer pilot in its namespace.

* Address comments from 7408 #7469

* allow users to configure cp namespace

when deploy the ingressgateway standalone

* rename cpNamespace to istioNamespace

* adding istioNamespace to values.yaml

* address shriram comment

reduce the need to add istio namespace in values.yaml

* delete accidentally added file

* simplify custom gateway yaml

chatting with shriram, i had trouble with my 2nd gateway working as it uses the same label as the first.  Thus attempt to simplify custom gateway.

* address costin's comment

* address shriram's comment

* dep ensure

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Gateway should use 8080 when mTLS disabled. (#7520)

* Implement resource multiplexing support in Galley. (#7458)

* Implement resource multiplexing support in Galley.

Many Mixer types live in different CRD Kinds, but map to the same
Proto type. To handle the case, this PR adds multiplexing support to
Galley.

Galley receives support for applying a converter to a resource, upon
reading. In Mixer's case, the converter converts the input CRD data to
an arbitrary proto type. In particular, the conversion code smuggles
Kind information through an inline field in a well-known wrapper proto
(i.e. LegacyMixerResource).

* Introduce a name field in LegacyMixerResource to smuggle in the name.
This is needed, as there are name collisions across different kinds
that map to LegacyMixerResource.

* Fix and re-enable mirroring e2e test (#7531)

* fix and re-enable mirroring e2e test

* fix mirror config file

* refactor airflow code (#7468)

* fix monthly pipeline to take a branch

* Add latest floating tag and replace hub&tag in values.yaml

* Add todo and fix file name

* Change floating tag name to match doc

* fix monthly pipeline

* simplify code, removed ordereddict

* add forgotten \

* remove pipeline_type for now

* finish rename of AirflowGetVariableOrBaseCase

* add modify_values.sh

* fix missing floating tag and fix dirty repo

* fix path

* add mofity_value step

* add global setting

* move daily common to its own file

* fix osx and win values.yaml

* refactor the code, add test

* use scripts instead of inline bash

* cleanup airflow scripts

* use airflow scripts everywhere

* fix monthly release

* squashed perf (#7539)

* Atomically create and start a test mock server (#7481)

* Atomic create and start a mock server

* Replaced NewServer(bool) with a StartNewServer()

* Removed always nil error handling

* Fix race condition in ControlZ. (#7576)

* Updated README for istio helm chart. (#7522)

Install CRDs first before install Istio.

* Adding an MCP backend to Mixer [Galley Integration] (#7433)

* Add a Mixer store backend for MCP protocol.

* call the gateway container of a gateway pod "istio-proxy" (#7566)

it is required for `istioctl proxy-config` tool to work correctly. `proxy-config`
looks for "istio-proxy" container, or for a single container in a pod.

When the gateway pod contains a single gateway container, proxy-config works.
Once an additional container is added, for any purpose, proxy-config will
stop working since it will not know from which container to
dump the config information.

Calling the contaner by the same name as the deployment, as it is today,
does not add much information. Calling the container `istio-proxy` makes it
clear that we have the same proxy in the gateway pod, as the sidecar proxy in
the application pods.

* Update codecov requirements (#7568)

* Update stale.yml (#7609)

* Update example adapter operator config to use CUMULATIVE. (#7614)

* Add ShellCheck job (#7373)

* Break excludes into many lines

* Add tools/run_shellcheck.sh

* Fix for usability by alpine's ash shell

* Add "shellcheck" job which calls tools/run_shellcheck.sh

* Add comment/link to shellcheck wiki

* Add file documentation

* Prow script for pilot-multicluster-e2e test (#7376)

* Prow script for pilot-multicluster-e2e test

Runs multicluster e2e test.  Will use mason to get GKE resources in
the same GCP project.  The resource_type will need to be updated
after the test-infra commit of the boskos config change to create the new
resource_type for multiple GKE clusters.

Script can be run with pre-allocated clusters by having a pre-setup
kubeconfig for multiple clusters.  It will choose the pilot cluster to
be the first cluster in the get-contexts or the PILOT_CLUSTER env var.

example:
PILOT_CLUSTER=gke_istio-ci-testing_us-east1-c_cluster-1 \
  KUBE_USER=tiswanso@cisco.com ROOT=$(pwd) USE_MASON_RESOURCE=False \
  prow/istio-pilot-multicluster-e2e.sh

* Adress review comments and fix HUB error w/ pilot e2e

* Merge code with updated e2e-suite

-Add functions for multi-cluster setup to cluster_lib

* Make cluster_setup work for multiple clusters. Setup clusterreg_dir.

* Setup firewall rule to allow intercluster pod to pod traffic.

* Remove post rebase resolved leftover e2e-common file.

* Remove use of instance-template for finding instance tags.

gcloud IAM roles seem to have instance-template.* as a higher privilidge than instance.list|get.

* Only attempt to set firewall rules for multicluster test.

* Setup service-account and build kubeconfig for pilot use.

* Make mason_lib set default gcloud project.

* Use the correct gcloud project and exec the v1alpha3 pilot test

* e2e multicluster fix review comments

* Use different serviceaccount name to avoid new conflict with helm istio-remote

* Increase timeout for multicluster test runs.

* Use sort | uniq to built gke cidr and nettag values for fw rule

- Also, fix some variable assignment style nits

* Update excludes for the shell scripts merged from release-1.0 (#7633)

* ShellCheck: Enable and fix SC2006 (#7632)

* Enable check for SC2006

* Fix SC2006 errors

* Properly format values.yaml for docification (#7644)

In the istio.github.io repository, we have a tool which processes
values.yaml.  In order for that tool to work correctly, every
level in the YAML must be 2 characters exactly.  In a few cases,
the spacing isn't two, but four, or alternately the lists have a prefix
of two spaces.  I'm not sure why helm processes this incorrect YAML, but
the parser we are using in the docs repo requires strict YAML.

* ShellCheck: Enable and fix syntax checking rules (#7648)

* Only check files with shebang on first line

* Fix if...then in install_prereqs_*.sh

* Enable syntax checking rules (1009, 1020, 1072, 1073)

* Fix SC2006 errors in install_prereqs_*.sh

* Fix multicluster e2e test setup to use helm chart template (#6679)

* Fix istio-remote and multicluster e2e test setup to use helm chart template

- Add istio-ingressgateway service and endpoints option to istio-remote charts
- Make the multicluster tests always setup istio-remote via helm template with
  passed args.
- Fix crash in getEndpointIPForService method

* e2e test install addons prior to remote cluster install

* Update e2e framework to work with istio-remote helm chart created configmap

- istio-remote changes remove selectorless service/endpoint setup so DNS
  for istio control plane services won't resolve on remote.  Sidecar
  injection configmap setup on remote uses the IP addresses of services
  directly.
  - change istioctl in e2e framework to allow use of sidecar inject
    configmap on remote rather than default settings.
- enable/disable sidecar injection in remote based on test flag
- misc. fix zipkin address in istio-remote configmap

* Multicluster pilot e2e: fix remote cleanup and ingress subtests

* ShellCheck: Enable and fix uncommon violations (#7649)

* Enable and fix SC2043

* Enable and fix SC2196

* Enable and fix SC2039 by upgrading sh to bash

* Enable and fix SC2028

* Enable and fix SC2154

* Enable and fix SC1128

* Enable and fix SC2021 - '[[:space:]]' -> '[:space:]'

* Enable and fix SC1113

* Enable and fix SC2007

* Enable and fix SC2128

* Enable and fix SC2126

* Enable and fix SC2120

* Enable SC2119

* Enable and fix SC2103

* Enable and fix SC2220

* Enable and fix SC2209

* Enable and fix SC2002

* Update SC2043 fix

* ShellCheck: Enable and fix SC2034 (#7671)

* Enable SC2034

* Remove unused '-m' flag in updateVersion.sh

* Remove unused '-i' flag in store_artifacts.sh

* Fix remaining SC2034 errors

* ShellCheck: Enable and fix SC2181 (#7636)

* Enable check for SC2181

* Rewrite minikube installation

* Fix SC2181 errors

* [helm chart] quotes for service annotation values (#7682)

* quotes for service annotation values

* add quote option to service annotations

* ShellCheck: Fix remaining minor violations (#7681)

* Enable and fix SC2124

* Break go build into multiple lines

* Fix SC2097 and SC2098 using a separate var for expansion

* Enable SC1054 and SC1056

* Enable and fix SC2129

* Enable and fix SC2004

* Enable and fix SC2048

* Enable and fix SC2148 by adding shebangs

* Enable and fix SC2100

* Enable and fix SC2230

* Enable and fix SC2231

* Enable and fix SC2145

* Enable and fix SC2001

* Enable and fix SC2035

* Enable and fix SC2059

* Enable and fix SC2162

* Enable and fix SC2166

* Enable SC1083

* Enable and fix SC2164

* Enable and fix SC2009

* Add config volumes and containers to gateways (#7654)

* add configVolumes to gateway pods

* add additionalContainers to the gateway

* Add common labels and release name to deployments (#7685)

* Add release name to generated yamls

* Common template to insert metadata labels to all deployments

* ShellCheck: Enable and fix SC2155 (#7670)

* Remove reassignment to SUCCESS_CASES[7]

* Enable and fix SC2155

* Fix and enable SC1117 (#7680)

*  use github.com/gogo/protobuf/proto for all Istio API resources (#7613)

* update vendor

* update galley

* update pilot

* update misc

* cleanup

* /bin/fmt.sh

* fix race in pkg/ctrlz package test coverage

* dep ensure

* remove duplicate unmarshalling code from mcp client

* Add query parameters for servicegraph to filter by destination and source workloads and namespaces (#6588) (#6812)

* Add ImagePullSecrets to Helm Chart components that don't use serviceaccounts (#7560)

* remove old code, move airflow scripts, fix build break (#7715)

* remove old code, move airflow scripts, fix build break

* comments

* [Ansible Installer] Including Missing SCC on change_scc file (#7708)

* Enable testlinter and fix violations of skip by issue rule. (#7697)

* Enable testlinter and fix violations of skip by issue rule.

* Add t.Skip() back to Test503sDuringChanges and Test503sWithBadClusters.

* merge commits https://github.com/istio/istio/commit/c3d749f0a1cc4fb7fdd76623bfdf9f1facd805e4, https://github.com/istio/istio/commit/7cf2320ef82526f9cf90a5c476c6de0f52e22662 from master (#7690)

* Add grafana persist option to istio helm chart (#7553)

On pod death, the current grafana chart does not persist any
configuration changes made in the UI (like adding alerts).
This adds a boolean option `grafana.persist` to create a
`PersistentVolumeClaim` that mounts to the existing grafana
data volume.

There is an additional option for `storageClassName` to override the
default storage class.

* Moved storageClassName to values.yaml instead of default operator

* Remove usages of mesh auth_policy flag (#7456)

* Remove usage of mesh auth_policy flag

* Remove out-of-date tests and comments

* Lint

* remove dead code in authn plugin. (#7552)

* remove dead code in authn plugin.

* update codecov requirements

* ShellCheck: Enable and fix quoting related rules (#7694)

* Enable and fix SC2016 by disabling instances of it

* Enable and fix SC2046

* Enable and fix SC2068

* Enable and fix SC2191

* Enable and fix SC2206

* Enable and fix SC2086

* Use an array for GOBUILDFLAGS

* Use arrays for cluster info

* Extract join_lines_by_comma function

* Use hostname instead of cluster name for SNI w/ istio mtls (#7732)

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* clean up (#7434)

Signed-off-by: Kuat Yessenov <kuat@google.com>

* Change Control-Z startup/shutdown interface to eliminate race. (#7705)

* bumps all the dep (#7714)

* Adding helper functions to simplify injection templates (#7737)

* Adding helper functions to simplify injection templates

* addressing comments

* Log error for bad values of autoInject (#7572)

* Add support for loadBalancerSourceRanges in gateways (#7744)

* Fix problematic `if`s and `while`s in shell scripts (#7702)

* Fix the dirty state in bin/get_workspace_status

* Fix other broken if-check

* Add missing '!'

* Remove redundant calls in while loops

* Use service name instead of cluster name for SNI string w/ Istio MTLS (#7692)

* Fix Gopkg.toml

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* use service name instead of cluster name for SNI

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Revert "Fix Gopkg.toml"

This reverts commit e90c95216f8f980dbc22926ae776bb8afc849fb1.

* lint and test fix

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* fix potential timer leak (#6964)

* remove deprecated flags (#7725)

* remove deprecated flags

* replace --v with --log_output_level

* Use per-push snapshot of configs for scalability (#7635)

* First draft of caching per-push protos

* Revert accidental change

* Remove lock - the structs are created before the push, in one routine - and read from multiple threads

* Fix tests

* Fix largest hotspot - GetServiceAttributes

* Format

* Round 3, optimizing DestinationRule

* Fix tests

* Remaining traces of ServiceAttributes

* Only parse as k8s if name ends with cluster.local. Hashmap for looking up service by host to be added later

* Add the namespace and service by name hashmap

* Initialize the map

* Move initialization to constructor

* Move initialization to constructor

* Found few more miss-uses of global push status

* Add allcaps comments around the global push status

* Move the initializaition of grpc server late - we were getting requests before init was completed

* It turns out it was crashing since we actually have 2 different environments

* Switch to the ready probe - endpointz can timeout in a large cluster

* Pilot plugins have 2 ways to pass env and common params - the struct and the method. Optimized initial connection initialization.

* remove uses of GetService

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* optimizations for virtual services

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* do not use hostname in auth plugin. use service

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Nix GetServiceAttributes

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Log GC, don't wait 30 sec after startup

* Debug shouldn't include all cache. TODO: separate debug status from context

* Adjust settings, use latest kiali

* Add a throttle for both pushes caused by 'all' and new connections.
Don't save snapshots of generated config by default.

* fix namespace hack in authn

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Add env to control push throttle and burst

* lint

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* PushStatus->PushContext

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* final lint

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Remove old throttle, and log how long connection was throttled

* Format

* Another easy one - we get proxy instances at the beginning of the call, no need to call n**3 times

* missed renaming

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* fix tests

Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* ShellCheck: Enable and fix SC1090 and SC1091 (#7748)

* Enable and fix SC1091

* Enable and fix SC1090

* Add comment tip for adding global excludes

* Changed the description font to small font (#7774)

* Change mixer_codegen.sh to use protoc dockerfile (#7343)

* Change mixer_codegen.sh to use protoc dockerfile

Add a dockerfile for counterfeiter and go-bindata

Add scripts for running counterfeiter and go-bindata using docker too

Add GOPATH env variable in counterfeiter.sh file

Updated ROOT calculation based on feedback

* Fix linter errors

* Add TODO with github issue in dockerfile generation script

* move mcp packages under common `pkg` directory (#7761)

* move istio.io/galley/pkg/mcp/* to istio.io/pkg/mcp/*

* update mixer mcp backend

* always create istio-sidecar-injector configmap with default install (#7565)

* always create istio-sidecar-injector configmap

Manual injection with kube-inject uses the sidecar template from the
in-cluster istio-sidecar-injector configmap. These configmap was not
created when automatic injection was disabled.

* add omitSidecarInjectorConfigMap for standalone gateway use case

* fixed:  duplicated port in kiali service (#7769)

* fixed:  duplicated port in kiali service

* changed port name from tcp to http-kiali

* Copy dump_kubernetes.sh to tools directory. (#6951)

* so that it could be included in the release archive.
* also include debug/authenticationz in its output.

* check cert/key not empty when applying tls setting. (#7610)

* check cert/key not empty when applying tls setting.

* address comment

* use shared informer to watch kubernetes resources (#7442)

* use shared informer

* run dep ensure

* istio-pilot shoud be in istio-system namespace (#7763)

* Custom proxy image for ingress, gateways, pilot and mixer. (#7784)

* speculate fix for flaky syncz tests (#7703)

* fixes for broken build (#7797)

* Defer self-registering validation webhook until endpoint is ready  (#7743)

* Wait for the istio-galley endpoint to show one ready address before
creating the validatingwebhookconfiguration. This works around the
lack of reconciliation in our current install process (e.g. kubectl
apply).

* fix linter

* update e2e tests

* typo

* build configuration if not present

* linter fix

* address review comments

* Run go generate on Istio repo from Istio root (#7788)

* Run go generate on Istio repo from Istio root

* Fix linting error

* Fix unit tests for check call where "connection.requested_server_name" is a new attribute we get from envoy
Fix gomatelinter error in adapter_test.go

* Missed some more cases for "connection.requested_server_name" in unit test..fixing that..

* Adding leftover authn_report_test.go files failing in unit test

* Fixing vendor/ files

* Fix quoting error with mixer_codegen.sh as per review

* Use debian-9 GCE image family. (#7816)

The debian-8 GCE image is deprecated so is not usable by default in the
deployment manager script.  So we'll bump up to debian-9 and that should
keep for a couple of years.

* Fixes for mesh expansion (#7789)

* default mtls

* Fix bug in generating certs with custom domain, and add the option for pilot

* Allow use of IP address (without having to change hosts or DNS).

Allow renewal or pilot or mixer certs with node agent - pilot may have
DNS SANa as well.

* Meshexpansion and multicluster default installer require mtls for connecting to pilot.

Make it default value.

* Use a startup script for node agent, so we can override settings.

* Add a way to run the script locally, as current user, without iptables.

This is useful for testing (as regular user)

* Make sure certs is owned by isto-proxy.

The new startup script for node agent is running as istio-proxy, to
avoid  root priv

* Add the 853 port - standard dns over tls

* Fix GCP and AWS as well, to allow IP address

* Fix GCP fetcher and startup flags

* Format

* Fix lint

* Rename failed to update one file

* Gencert supports generating cert signed by citadel. (#7716)

* Gencert supports generating cert signed by citadel.

* use constant var for default value.

* Address comments.

* fix lint.

* Add docker build change.

* Change to ALPN based filter chain selection. (#6612)

* Change to ALPN based filter chain selection.

* Address the comments.

* Rename as purpose, change build tls context logic.

* Remove allowTLS test case, now superseded by multi-filter-chain.

* Move to util, and fix the comments.

* Always set the filter chain match, but it doesn't work.

* Change to setup filter chains match and tls context together.

* Add tls_inspector listner filter, unclear why it works without it.

* Fix the ALPNInMesh.

* Lint fixing.

* Remove stale comments and code.

* Add OnFilterChains callback to the plugin.

* Add Plugin.OnFilterChains method.

* WIP modifying authn plugin unit test.

* Get unit test work.

* WIP for the changing OnFilterChains signature.

* fix the fake plugin interface.

* Add TODO to finish the new OnFilterChains consuming.

* multiplexing config is correct, rest needs testing.

* Remove some dead code.

* Rename to OnInboundFilterChains.

* Fix the match and authn unit test.

* Remove dead comment.

* OnInboundFilterChains cover TCP case as well.

* Remove debugging print.

* Address comments, requiretls refactored.

* fix lint.

* Change how to call GetConsolidateAuthenticationPolicy

* Remove some unintended old code.

* Append all filter chains.

* Add a todo for alpn values in LDS.

*  Merge shellcheck job with lint job (#7785)

* Add .circleci/lint.dockerfile (adds shellcheck)

* Merge shellcheck job with lint job

* Normalize .circleci/Makefile

* Add recipes for lint image

* more build fixes related to shell check (#7815)

* more shellcheck fixes

* fix make targets

* fix make format

* run dep prune to cleanup unused files (#7799)

* Replace "MaxMsgSize" with "MaxRecvMsgSize" in grpc server (#7796)

* Remove blank excludes flag (#7809)

* Change mixer_codegen.sh script to use gogoslick and docs gen from protoc docker file directly rather than building it in the script. (#7824)

Also excluded files generated using counterfeiter from getting formatted through bin/fmt.sh file

* groups standard golang libs (#7800)

* [Ansible Installer] Adding Routes to Prometheus, Grafana, Istio-Ingress-Gateway (#7741)

* [Ansible Installer] Change the order of delete resources

* [Ansible Installer] Adding Routes to Istio Ingress Gateway, Prometheus
and Grafana in Openshift

* Disable Script Check in CircleCI as it is failing jobs. (#7862)

* Merge collab-test-framework into master (#7739)

* [test-framework] Design sketch for test framework.

* [test-framework] Reorgnanization and linter fixes (#5128)

* [test-framework] Adding pilot test showcase (doesn't build)

* [test-framework] Adding mixer test showcase (doesn't build)

* [test-framework] Get the code to build.

* [test-framework] Cleanup the dependency model. (#5212)

* Cleanup the dependency model.

+ Colocate all (current) dependencies.
+ Tighten the Dependency interface.
+ Introduce internal.Stateful to reduce the surface area of Dependency.

* Cleanup and linter fixes.

* [test-framework] Adding basic cluster app (#5257)

* [test-framework] Adding cluster App creation (#5342)

* [test-framework] Add example to simple showcase test (#5369)

* add new changes to simple showcase test

* Fix typo

* [test-framework] Rationalize FortioApp model. (#1) (#5452)

* Rationalize FortioApp model.

- GetFortioApp returns a single instance of DeployedFortioApp and error.
It uses a name pick the app directly, similar to the GetApp model.
- Add *OrFail overload, similar to GetApp case.
- Add GetFortioApps which has a label selector, and returns an array of
deployed apps.

* More cleanup of the Fortio code and get the code to compile.

* [test-framework] First pass cleanup of the model (#2) (#5451)

* First pass cleanup of the model:

- Add CallOrFail to DeployedApp for expedient test authoring.
- Replace the Mixer mock adapter with a mock PolicyBackend. This is much
easier to manage from a testing standpoint.
- Remove label based custom logic. Labels are used for filtering only.
- Align Mixer showcase test with the Pilot app model. This way, we can
author Mixer integrations tests by applying configuration and controlling
app/Mixer backend behavior.

* Accommodate CR feedback.

* [test-framework] Remove charts/DeployedIstioComponent and Environment cleanup (#5474)

* Remove charts and DeployedIstioComponent.

* Cleanup the Environment interface and enforce methods in local and
cluster implementations.

* [test-framework] Refactor the driver interface and implementation. (#5475)

* [test-framework] Cleaning up App API to simplify URL construction (#5487)

* [test-framework] API and command-line cleanup. (#5510)

* Improve the command-line handling and remove ApiServer & Helm code.

* - Check double-run in driver.Interface.Run().
- Add support for suite level dependency processing.
- Use scoped logs in the driver.

* [test-framework] Add internal interface for the environment (#5523)

* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* [test-framework] Adding bootstrap for local Envoy and Pilot App (#5513)

* [test-framework] Add health check to local envoy agent at startup (#5551)

Also fixing a bug in the envoy yaml template that was preventing the
configuration of multiple ports to a single instance.  Expanded the
test to cover this.

* [test-framework] Switch from testing.T => testing.TB and uniformize *OrFail. (#5716)

* [test-framework] Further rationalize driver model. (#5742)

* Further rationalize driver model.

- Extract out a "Context" to be used internally within the testing code. This is mainly useful to break cyclic dependency between the environment and the driver code.
- Cleanup the logging code: Most of the code here is not needed anymore: as the standard set of logging flags already take care of writing logs to well known locations
- Similarly, tmp code can be cleaned up now, to mainly focus on per-component state storage. Moved the code within driver folder to reduce fragmentation.

* Move the GetEnvironment method from the driver interface to Context.

The move allows us to use context object as the single handle to pass
through the rest of the test-framework stack.

* Minor naming fix.

* [test-framework] Revamp the internal machinery of the test framework. (#5804)

- Simplify the dependency model. Instead of dependencies having logic, they simply are enums.
- Rename GetEnvironment to AcquireEnvironment and change semantics to at most one per test. With this model, we can reset the internal state of dependencies during the acquisition of the environment.
- Implement a loose object model that opt-in to test framework services through interface implementation. The three services that can be opt-in to are: receiving user supplied configuration, cleanup, and reset with every environment acquisition.
- Implement a resource tracker for supplying these services.
- Switch to a common, internal environment interface, as opposed to environment specific internal interfaces. This switches the dispatch model so that environments are inherently aware of how each dependency work, rather than dependencies adjusting themselves for each environment. In effect, the internal structure of the code should be simplified.
- Use os.TempDir() as the default workdir.
- Fix formatting of help output.
- Switch internal.TestContext interface to be a struct to reduce code clutter.

* [test-framework] Implement Mixer related infrastructure for local environment. (#5805)

* Add local-Mixer infrastructure and implement a basic test to cover.

* Accommodate CR feedback.

* Separate out the settings.

* [test-framework] Adding local pilot (#6077)

* [test-framework] Add cluster support to the test framework. (#6179)

* Add cluster support to the test framework.

* CR feedback.

* Minor linter fixes.

* [test-framework] Integrating local Pilot and Envoy (#6332)

- Updated configuration of Envoy to use discovery v2 API on Pilot

- Adding a discovery proxy to allow interception of responses from
Pilot. This will provide the ability to modify the Envoy configs to
support local testing.

- Updated the agent test to incorporate a local pilot. Also starting
to hash out how to determine whether or not an Envoy has received
a configuration update.

* [test-framework] Refactoring pilot test application (#6489)

Simplifying the structure to make the code more readable and to make it
better fit in with the application model of the new local agent.

* [test-framework] A few fixes for local envoy code (#6537)

* [test-framework] More robust port reservation (#6736)

This PR reserves a block of ports, so that future attempts to pick
a port will not collide with previous attempts.

* [test-framework] Add control RPC to test service (#6737)

The RPC allows us to simulate in-mesh traffic from one service to
another.

Creating a copy of the pilot test service under pkg/test. This is a
more appropriate home for the code and also avoids changing the rest
of the code that depends on the old service.

* [test-framework] Support local in-mesh traffic (#6743)

There are a lot of changes here that are interrelated.  Here's the summary.

1) Adds concept of a protocol client, which is used to intercept outbound
URLs. The interception is performed by the proxy which modifies the
URL host:port so that an outbound call from service A to B will first go
through service A's Envoy proxy.

2) Moved the discovery response filtering logic into the application proxy,
since the additional logic requires state from the proxy.

3) Updated the discovery filtering logic to intercept the outbound listeners
created by Pilot to enable communication with other services
(e.g. so A can talk to B). Pilot uses virtual listeners which are not
actually bound to a port (since outbound traffic would typically be
redirected to 15001 in a cluster configuration). The updated logic
modifies these listeners by assigning a newly reserved port and forcing
the listener to bind to that port. The new port is then stored in a
port map, which is used to modify the URLs on outbound traffic from the
application.

* [test-framework] Remove tagging/label support. (#6975)

* Remove tagging/label support.

* remove ignore

* [test-framework] Add API Server support. (#7276)

* [test-framework] APIServer + Galley tests (#7277)

* [test-framework] Code cleanup: Move and simplify the core driver code. (#7286)

* Refactor the core driver code:

+ Created test/framework/ as the namespace of the internal/implementation
of the test framework.
+ Moved the top-level driver code to test/framework/driver
+ Simplified the code at the pkg/test level, leaving only framework.go
for top-level test framework operations.
+ Simplified code in framework.go to simply redirect to the driver.

* More code move:

+ Created framework/environments and moveed cluster & local there.
+ Renamed cluster to kubernetes.
+ Moved test/internal to test/framework/internal.
+ Moved test/tmpl to test/framework/tml.

Also renamed environment.Interface to framework.Environment.

* More shuffling:

+ Moved test/dependency =? test/framework/dependency.
+ Moved framework/environment.go => framework/environment/environment.go
This is needed to break the dependency cycle.
+ Moved environment variable flags, args code to framework/settings.
+ Moved test/framework(formerly operations.go) to test/api.go.
+ Moved driver code from framework/driver/... to framework/

* Fix the mutex error.

* Rename construct => new

* [test-framework] Support non-HTTP ports in local agent (#7419)

Also doing some other cleanup and build fixes.

* [test-framework] Componentize the test framework. (#7344)

* Refactor and componentize the test-framework.

* Accommodate CR feedback.

* [test-framework] Move pilot to components. (#7515)

* [test-framework] Some minor cleanup for pilot component (#7536)

* [test-framework] Adding support for component dependencies (#7598)

- Added Component interface, which is now implemented by all components

- Separate component registries for local and kubernetes

- Updated the dependency Tracker to initialize all dependencies of
a component before initializing the component, itself.

* [test-framework] Integrating test apps into framework (#7628)

* [test-framework] Updating jsonpb dependency

* [test-framework] fixing linter errors in generated files

Needed to rename the files so they'd be ignored.

* [test-framework] Removing t.Skip calls in unit tests.

* [test-framework] removing accidental edit

* more fixes for broken daily build (#7839)

* more fixes for broken daily build

* fixes shell lint

* Add option to deployableConfig to remove existing resources (#7778)

* Add option to deployableConfig to remove existing resources

* Fix typos

* Fix bad assign

* Reviews

* mixerplugin: test for mTLS mixer attributes (#7791)

* stash

Signed-off-by: Kuat Yessenov <kuat@google.com>

* incremental files

Signed-off-by: Kuat Yessenov <kuat@google.com>

* rebase

Signed-off-by: Kuat Yessenov <kuat@google.com>

* mtls test

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fixing tests

Signed-off-by: Kuat Yessenov <kuat@google.com>

* fix tests

Signed-off-by: Kuat Yessenov <kuat@google.com>

* linting

Signed-off-by: Kuat Yessenov <kuat@google.com>

* use istio/istio issue

Signed-off-by: Kuat Yessenov <kuat@google.com>

* disable tests

Signed-off-by: Kuat Yessenov <kuat@google.com>

* try skipping udp

Signed-off-by: Kuat Yessenov <kuat@google.com>

* Add periods at the end of description strings. (#7668)

* Rename updater update to apply (#7786)

* Rename updater update to apply

- This rename allows to combine both updater and ConfigStoreCache

Co-authored-by: Utako Ueda <uueda@pivotal.io>

* Rename missing comment

Co-authored-by: Utako Ueda <uueda@pivotal.io>

* Fix corresponding structs that implement update

Co-authored-by: Utako Ueda <uueda@pivotal.io>

* RBAC: handle all request.auth.claims[] using RBAC list matcher (#7747)

* RBAC: handle all request.auth.claims[] using RBAC list matcher

* Update the go-control-plane dependency

* Refactor generateMetadataListMatcher()

* Add a test of groups in TestConvertRbacRulesToFilterConfig()

* Fix lint error

* Refactor createDynamicMetadataMatcher()

* Remove redundant local files

* Fix dependency error

* Change request.auth.groups to request.auth.claims[groups]

* Change based on the review comments

* Organize groups claim under auth.derived.claims

* Change based on the review comments

* Handle all request.auth.claims[] using RBAC list matcher

* Fix a typo and a lint error

* Change based on review comments

* use rate.Limiter to limit the rate of operations (#7802)

* /watch has been deprecated, use `?watch=true` (#7443)

* e2e guide docs typo fix (#7906)

* tests/e2e/README.md typo fix

* tests/e2e/UsingGKE.md typo fix

* ix spelling error in file tests/util/checker/testlinter/README.md (#7898)

* xds: fix invalid comments (#7892)

* fix invalid comments

* address comments

* Cloudfoundry - Route weight (#7868)

* bumps cloudfoundry copilot dependency

Co-authored-by: Kauana dos Santos <kdossantos@pivotal.io>

* adds support for copilot sending weights

Co-authored-by: Kauana dos Santos <kdossantos@pivotal.io>

* fix typo in snapshot log/comments (#7853)

* istioclt -> istioctl in help string (#7846)

* lock protect read adsClients (#7845)

* lock protect read adsClients

* remove redundant lock

* Add RDS load timestamp to istioctl proxy-status (#7833)

This commit adds the RDS load timestamp to istioctl proxy-status <pod>
i.e. by obtaining the most recent LastUpdated value from all available
DynamicRouteConfigs.

Signed-off-by: Venil Noronha <veniln@vmware.com>

* More sidecar injector template cleanup (#7832)

* More sidecar injector template cleanup

* fixing

* optimize `edsClusters` `connectionNumber` to reduce access contention (#7842)

* optimize edsClusters to reduce access contention, and add protection when get its length

* address comments: use atomic.AddInt64 return value to prevent load

* remove useless param (#7904)

* Helm chart: allow annotations on mixer pods (#7557)

* Experimental dual-use mode for citadel (#7629)

* experimental dual-use mode for citadel

* limit dual-use CN to single host

* validate common-name length

* Add loadBalancerIP/loadBalancerSourceRanges (#7686)

* store_artifacts.sh: inject LICENSES.txt file into all Docker images (#7712)

* remove unused post-install hook. (#7717)

* Fix path of manifest file for bookinfo sample application (#7753)

*  fix pilot crash issue if there is zero listeners and err is nil (#7801)

* fix pilot crash issue if there is zero listeners and err is nil

* fix pilot crash issue if there is zero listeners and err is nil

* Update install_addons.yml (#7829)

* remove unused post install hook for remote chart. (#7718)

* Mark agent_test tests skipped, as they are failing in CircleCI. (#7911)

* Add ControlZ support to Citadel. (#7704)

* Revert "remove unused post install hook for remote chart. (#7718)" (#7914)

* Revert "remove unused post install hook for remote chart. (#7718)"

This reverts commit 4a78161824da9c7712a07f245487316c689b0245.

This seems to be the potential culprit in some of the postsubmit jobs
and checkin gates.

* Revert "remove unused post-install hook. (#7717)"

This reverts commit 1d62f44a1a7a2e0cfe2c9381f9e1ad53a7eb9fdb.

* Unify reconnection and message processing in mcp client. (#7811)

* Unify retry and processing logic in mcp client.

Merge the (re)connection and processing logic into a single function
and unify the error handling and retry code paths. Update the ctrlz
tests to account for the fact that the initial MCP requests are now
captured in the journel.

This also adds unit tests for the reconnection logic on send and
receive failures.

* remove extra debug code

* fix race in pkg/mcp/configz unit tests

* linter fixes

* update comments

* add deleted line

* Enable permissive mode for default deployment (#7750)

* Enable permissive mode for default deployment

* Add more text

* fix format

* Fix test

* fix small lint

* Address lint

* Fix error

* Fix lint

* Fix lint

* Fix lint

* Fix test

* Revert unrelated change

* Rebase the change

* Fix change

* more fix

* Implement mTLS functionality in MCP (#7830)

* Implement mTLS functionality in MCP.

Add utility code to load&watch certificates and build a TransportCredentials object.
Add an authenticaton check mechanism to admit/fail incoming streams.
Add a basic list-based implementation for authentication check.
Move test certificates from galley/validator folder to pkg/mcp/testing.

* Accommodate CR comments
#1

* [test-framework] Remove use of Cobra flags. (#7887)

* [test-framework] Remove use of Cobra flags.

The way they were used was not working properly in GoLand.

* addressing comments.

* Implement basic mTLS functionality in Galley (#7881)

* Implement basic mTLS support in Galley.

* Add warning todo for symlinks.

* Accommodate downstream changes.

* Accommodate review comments.

* lint tools complain about redundant newline in Println (#7929)

* validate updated value is allowed (#7932)

* Fix RedisQuota not returning the first override match (#7913)

Redisquota currently returns the last match which is against the expectations

* Re-enable "Check Scripts" as optional part of make lint (#7886)

* Ensure shellcheck is installed

* Fix typo in comment: excludes -> exclude

* Revert "Merge shellcheck job with lint job (#7785)"

This reverts commit ffa47868a802b14dc24220041c52bf6ffdd3943f.

* Add Galley server mode to Helm charts. (#7882)

*  Stop using github.com/bradfitz/slice.Sort (clone of #7775 with go dep fixes) (#7819)

* Stop using github.com/bradfitz/slice.Sort.

It's in Go's sort package as of Go 1.8, and Istio depends on Go 1.9
(docs) or Go 1.10 (CircleCI config). In any case, stop using my
package, which isn't updated for new architectures now that it's in
Go's std.

Background: I received a bug report that Istio didn't build on
somebody's non-x86 big endian architecture. (Bug report didn't say
which)

* dep snreu

* make depend.diff

* de-dup clusters by name (#7939)

* Update grafana to its latest version (#7945)

* Update grafana to its latest version

https://github.com/istio/istio/blob/master/addons/grafana/Dockerfile.grafana#L1 shows
that Istio currently uses the version 5.0.4 of grafana, which was released on March 27, 2018.
The latest version of grafana is 5.2.2, released on July 24, 2018. Istio should update its
grafana version to leverage the fixes included in the latest grafana version.

* Fix the error on "make docker.grafana"

* add rbac attributes to attributemanifest (#7919)

* add rbac attributes to attributemanifest

* address comment

* Helm chart: allow mapping for statsd-prom-bridge (#7011)

* Update fortio to 1.1.0 (#7871)

Note that running ` dep ensure --update istio.io/fortio`
Changed not just what is in this PR but a bunch of other files that I
excluded… someone needs to see why vendor seems out of sync with the
Gopkg.*

* Support configurable domain in Citadel identities (#7849)

* document domain as configurable, remove 0.2 reference

* add support for configurable domain in  SPIFFE URL

- add command line option to specify domain (defaults to "cluster.local" for backward compatibility)
- add domain to controller and use in generated names
- add to test calls

* add support for configurable domain in SPIFFE URL

- add command line option to specify domain (defaults to "cluster.local" for backward compatibility)
- add domain to controller and use in generated identity names
- add to test calls
- add to security helm chart and values

* set default domain instead of failing

* fixed indentation and formatting

* merge sanfu revived defaultIdentityDomain...

* Revert "remove requestedServerName attribute (#7278)" (#7981)

This reverts commit 31398b36c17733af015dbcfe55f999e2c51a096e.

* Updated Jaeger to 1.6 (#7970)

* Support Timestamp type for valueTypeEncoder (#7924)

* Support Timestamp type

An expression type like `response.time` is TIMESTAMP, so add new encoder
 to encode `time.Time` as `v1beat1.Timestamp`.

* Update issue templates to make 'em look nicer for the submitter. (#7999)

* Add WaitForSync to move up the retry loop (#7960)

Signed-off-by: Kuat Yessenov <kuat@google.com>

* Update link to more options for E2E test (#6710)

Create ContinuousDevelopment.md

Add link to ContinuousDevelopment Doc

Update README.md

Update README.md

Create Tips

Rename Tips to Tips.md

Delete ContinuousDevelopment.md

Update README.md

Update README.md

* remove destination service (#8007)

Signed-off-by: Kuat Yessenov <kuat@google.com>

* Let "pilot_grpc_address" take over param DiscoveryAddress in WriteBootstrap (#7900)

* Let "pilot_grpc_address" take over param DiscoveryAddress in WriteBootstrap

* update bootstrap test

* update install templates

* replace MaxClusterNameLength with MeshConfig.StatNameLength (#7905)

* Fix current ShellCheck violations (#7948)

* Use an array for MAKE_TARGETS

* Use IMPORTS as an array

* Use DESCRIP…