Skip to content

Conversation

ostromart
Copy link
Owner

This is just a quick and dirty POC to see if this if going down the right path. Looking for high level feedback.

Nathan Mittler and others added 30 commits February 14, 2018 15:35
Automatic merge from submit-queue.

Experiment for improving pilot handling of large services

The throttle on event processing is removed by default, can be added back if needed (but so far my tests show it's better without). By processing events faster there is less cache recomputation, for a given sidecar QPS. By prolonging the update, more cache clear were happening and more requests from envoy with an empty cache.

Also added logs, to evaluate how many requests from envoy and what the latency is. The sidecar for pilot doesn't report mixer metrics, and pilot doesn't log response times.
* Add integration tests to verify Mixerclient stats.

* Update proxy SHA to latest.

* Disable report batch to fix test flakiness.
…stio#3497)

Automatic merge from submit-queue.

x509 error with certificate-authority-data: pin mergo dependency against kubernetes/client-go compatible version

This issue was noticed when istioctl returned
"x509: certificate signed by unknown authority"
in combination with a cluster in ~/.kube/config that
specifies its cluster certificate using certificate-authority-data.

The merging in
https://github.com/kubernetes/client-go/blob/b6a34c5a002893138005771f76480e7166da9310/tools/clientcmd/loader.go#L225-L227
results in `CertificateAuthorityData` containing the base64-decoded
certificate twice, which results in the x509 error.

Potentially related to https://github.com/kubernetes/client-go/blob/7cd1d3291b7d9b1e2d54d4b69eb65995eaf8888e/tools/clientcmd/client_config.go#L160-L162.
Not sure if that's the right way to fix that issue, I'd also appreciate
some pointers on the best approach to add tests for this.
Automatic merge from submit-queue.

Add ON-CALL playbook

replaces https://goo.gl/9xUCRB and https://goo.gl/Hrg94p
* Update dev guide file

* Update dev guide file
* Add flexvolume driver to istio security.

* Add basic tests and harness for flexvolume testing.

* Add more negative tests

* Add credential create/delete failure UT.

* Fix up linter issues.

* Fix typo.

* Fix lint issues.

* Address review feedback.

* Fix formatting.

* Adress review comments.

* Fix file permissions.
Automatic merge from submit-queue.

[DO NOT MERGE] Auto PR to update dependencies of istio

This PR will be merged automatically once checks are successful.
```release-note
none
```
…stio#3492)

Automatic merge from submit-queue.

Update .gitignore and make clean. Fix namespace in install template.
* Revise reeadme file

* Fix typo
Automatic merge from submit-queue.

Correctly supporting auth modes in pilot e2e tests
* (Rfc) adding vendor as submodule

Receiving objects: 100% (4145/4145), 8.26 MiB | 930.00 KiB/s, done.

Note: adding pruning instruction to Gopkg.toml makes vendor be 54Mb
instead of 407Mb

* Example of rebase, matching master's change to .lock

* Adding submodule target to init + example of update flow

Ran
dep ensure --update istio.io/fortio
To pick up 0.6.8 from 0.6.7
Checked in vendor/ submodule first and now this commit

* No need to cache vendor with this setup

* No cache needed/used

* Added make targets like fortio's for submodule

+ Fun of touching dozens of circleci builds...

* Removed unnecessary .lock copy

* Merge and dip ensure

* Fix for istio#3483

* Update from master

matches master ed69c3a and
istio#3497
Automatic merge from submit-queue.

Only use output struct values when actually generated

This PR is an approach to improving data quality for data generated by preprocess
adapters. The PR updates the generated code for Output structs for preprocess adapters
by adding SetXXXX() methods to the Output type that keep track of attributes that
were set. The generated code that checks for attribute values then also checks to see
if an attribute value was set.

In support of those changes, this PR adds a new expr Fn to Mixer: `emptyStringMap()`. This
new expr allows creation of default values for string_map expressions.

Finally, the PR updates the various config components for kubernetesenv adapters throughout
the repo.

This should eliminate the `source.service == ""` data issues that appeared when the new preprocessor
adapter mechanisms were put in place (as well as other zero-value related issues).
Automatic merge from submit-queue.

Make IsNetworkFilterConfig exported

Before this change, it is hard to import the v1 API structures and define
a new custom NetworkFilterConfig in an external module.  A NetworkFilterConfig
needs to implement isNetworkFilterConfig, but that method is private to the v1
module.  This change makes it exported.

This is useful for people who want to write Pilot webhooks that mutate
the xDS data---the relevant data structures are already defined.

Signed-off-by: Spike Curtis <spike@tigera.io>
* Adding simple e2e test to presubmit

* Update e2e-simpleTests.sh
Automatic merge from submit-queue.

Add the tunning variable for throttle

Add the env to the template - will need to be adjusted for large clusters, memory 
pressure is still pretty high and may go above k8s limits.
Automatic merge from submit-queue.

[DO NOT MERGE] Auto PR to update dependencies of istio

This PR will be merged automatically once checks are successful.
```release-note
none
```
Automatic merge from submit-queue.

Remove obsolete mixer deployment files.

Up to date deployment files are under install/kubernetes. These are
out of date and cause confusion.
Automatic merge from submit-queue.

Fix Grafana graphs by filtering the unknown service.

The new behavior is that an "unknown" service is logging metrics as ingress to `istio-ingress.istio-system`. This filters that.

Now `istio-ingress` shows up as it's own service w/ graphs, but some don't work with the traffic coming from "unknown". I'll keep working on that, but at least this change fixes the whole page not working from the "unknown" service showing up in the selector at the top.
Previously it accepted context.Context for the validity of the
store, but that does not fit well with the mixer's calling
structure. Also context's document says a struct shouldn't store
the context as a field.

This PR adds Stop method to do this.
* Squash clear cache calls

* Format/lint

* Missed return..

* Reduce squash to 1 sec

* Revert istio version

* Format..

* Fix the test - cache will no longer reset immediately

* Make the cache squash configurable, to evaluate impact

* Fix formatting
Automatic merge from submit-queue.

Upgrade istio-builder image to 0.4.6

See discussion in istio#3279

/assgin @mattdelco @sebastienvas @rkpagadala
Automatic merge from submit-queue.

Activate VM expansion test
Automatic merge from submit-queue.

IP Whitelist in Helm

I added an option to specify `includeIPRanges` in the helm chart in order to match the corresponding argument to `istioctl` when using automatic sidecar injection. 

I also fixed an issue with the value name for the sidecar injector where it didn't match the name of the sidecar injector subchart, and thus the values weren't available to it.
mandarjog and others added 19 commits February 17, 2018 18:46
Automatic merge from submit-queue.

Re-enable pprof for mixer with args consistent with Pilot

Enable pprof at http://localhost:9093/debug/pprof/* with command line args consistent with Pilot.

Signed-off-by: Mandar U Jog <mjog@google.com>
* vendor_update_Feb17_2018

Ran dep ensure —update

Also bump up fortio version

Fixes istio#3585

* Update to head

istio/old_vendor-istio_repo#2 is merged

Ran dep ensure
A bit surprised the api SHA change didn’t trigger any file change
istio/api#376 (comment)
Automatic merge from submit-queue.

More permissive routing tests

Adds retries and makes routing check less strict. Should reduce flaky test failures while still verifying the behavior.
Automatic merge from submit-queue.

use current time for running daily release
Automatic merge from submit-queue.

Fix CA main function format.

Rename NewIstioCAOptions to NewPluggedCertIstioCAOptions.
Automatic merge from submit-queue.

[DO NOT MERGE] Auto PR to update dependencies of istio

This PR will be merged automatically once checks are successful.
```release-note
none
```
* Refine the node agent logic

* fix lint
Automatic merge from submit-queue.

Change mock services to use separate port object

Before, HelloService and WorldService mock sharing the PortHttp object. This cause incorrect behavior when modifying this port for testing (e.g adding auth annotation), as the change will be applied to both services instead of just the desired one.
Automatic merge from submit-queue.

[DO NOT MERGE] Auto PR to update dependencies of istio

This PR will be merged automatically once checks are successful.
```release-note
none
```
Automatic merge from submit-queue.

more lint fixes

noticed that "tests" is specified in lintconfig_base.json incorrectly.
It's "test" actually.

By changing this, I've met some new lint errors in test.go files,
mostly on errcheck. This PR includes fixes on them.

Note: I've seen several patterns of `defer something.Close()`,
which does not check the errors. I think this is very common among
our code base, and I don't think this has problem if that's within
test code. I added an exclude pattern for this.
* Clean up the platform config

* Fix lint
Automatic merge from submit-queue.

Disable vm test in cluster wide

Cluster provisioning is enabled on cluster wide test, and vm test has not been updated to use the pre-provisioned VM and is creating instead a VM in the main testing project. The VM test is using local network IP, and the cluster and the VM are not in the same project which makes communication between those 2 impossible.

Disabling the vm test for now until we can take advantage of the provisioned VM
@ostromart ostromart closed this Feb 21, 2018
@ostromart ostromart deleted the istio_metrics_ostromart branch February 21, 2018 00:54
ostromart pushed a commit that referenced this pull request Apr 20, 2018
* Vendor changes adding Informers and Listers

* Secret Controller code

* Linter detected issues

* Vendor update related sha change

* Adding final bits to the controller

* Fixing controller startup code

* Adding required RBAC rules to watch for secrets

* Refactor Cluster Store initialization place

* Fixing Unit test failure

* Fixing Unit test failure

* Addressing comments part #1

* Fixing Unit test

* Switching to different type of Informer

* Add create k8s_cr.Cluster object

* Fixing if statement

* Fixing lint error

* Cosmetic changes

* Fixing lint error
ostromart pushed a commit that referenced this pull request Jun 23, 2018
* # This is a combination of 5 commits.
# This is the 1st commit message:

update istio.io/api for new mcp definitions

# This is the commit message #1:

update mcp server

# This is the commit message #2:

update snapshot package

# This is the commit message #3:

use mcp service in galley/pkg/server

# This is the commit message #4:

fix linter

* update istio.io/api for new mcp definitions

update the latest mcp service definitions from istio.io/api

* fix linter error (again?)
ostromart pushed a commit that referenced this pull request Aug 17, 2018
* [test-framework] Design sketch for test framework.

* [test-framework] Reorgnanization and linter fixes (istio#5128)

* [test-framework] Adding pilot test showcase (doesn't build)

* [test-framework] Adding mixer test showcase (doesn't build)

* [test-framework] Get the code to build.

* [test-framework] Cleanup the dependency model. (istio#5212)

* Cleanup the dependency model.

+ Colocate all (current) dependencies.
+ Tighten the Dependency interface.
+ Introduce internal.Stateful to reduce the surface area of Dependency.

* Cleanup and linter fixes.

* [test-framework] Adding basic cluster app (istio#5257)

* [test-framework] Adding cluster App creation (istio#5342)

* [test-framework] Add example to simple showcase test (istio#5369)

* add new changes to simple showcase test

* Fix typo

* [test-framework] Rationalize FortioApp model. (#1) (istio#5452)

* Rationalize FortioApp model.

- GetFortioApp returns a single instance of DeployedFortioApp and error.
It uses a name pick the app directly, similar to the GetApp model.
- Add *OrFail overload, similar to GetApp case.
- Add GetFortioApps which has a label selector, and returns an array of
deployed apps.

* More cleanup of the Fortio code and get the code to compile.

* [test-framework] First pass cleanup of the model (#2) (istio#5451)

* First pass cleanup of the model:

- Add CallOrFail to DeployedApp for expedient test authoring.
- Replace the Mixer mock adapter with a mock PolicyBackend. This is much
easier to manage from a testing standpoint.
- Remove label based custom logic. Labels are used for filtering only.
- Align Mixer showcase test with the Pilot app model. This way, we can
author Mixer integrations tests by applying configuration and controlling
app/Mixer backend behavior.

* Accommodate CR feedback.

* [test-framework] Remove charts/DeployedIstioComponent and Environment cleanup (istio#5474)

* Remove charts and DeployedIstioComponent.

* Cleanup the Environment interface and enforce methods in local and
cluster implementations.

* [test-framework] Refactor the driver interface and implementation. (istio#5475)

* [test-framework] Cleaning up App API to simplify URL construction (istio#5487)

* [test-framework] API and command-line cleanup. (istio#5510)

* Improve the command-line handling and remove ApiServer & Helm code.

* - Check double-run in driver.Interface.Run().
- Add support for suite level dependency processing.
- Use scoped logs in the driver.

* [test-framework] Add internal interface for the environment (istio#5523)

* Minor improvements to environment & dependency interfaces.

* Add internal environment interfaces.

* Fix formatting and some linter issues.

* [test-framework] Adding bootstrap for local Envoy and Pilot App (istio#5513)

* [test-framework] Add health check to local envoy agent at startup (istio#5551)

Also fixing a bug in the envoy yaml template that was preventing the
configuration of multiple ports to a single instance.  Expanded the
test to cover this.

* [test-framework] Switch from testing.T => testing.TB and uniformize *OrFail. (istio#5716)

* [test-framework] Further rationalize driver model. (istio#5742)

* Further rationalize driver model.

- Extract out a "Context" to be used internally within the testing code. This is mainly useful to break cyclic dependency between the environment and the driver code.
- Cleanup the logging code: Most of the code here is not needed anymore: as the standard set of logging flags already take care of writing logs to well known locations
- Similarly, tmp code can be cleaned up now, to mainly focus on per-component state storage. Moved the code within driver folder to reduce fragmentation.

* Move the GetEnvironment method from the driver interface to Context.

The move allows us to use context object as the single handle to pass
through the rest of the test-framework stack.

* Minor naming fix.

* [test-framework] Revamp the internal machinery of the test framework. (istio#5804)

- Simplify the dependency model. Instead of dependencies having logic, they simply are enums.
- Rename GetEnvironment to AcquireEnvironment and change semantics to at most one per test. With this model, we can reset the internal state of dependencies during the acquisition of the environment.
- Implement a loose object model that opt-in to test framework services through interface implementation. The three services that can be opt-in to are: receiving user supplied configuration, cleanup, and reset with every environment acquisition.
- Implement a resource tracker for supplying these services.
- Switch to a common, internal environment interface, as opposed to environment specific internal interfaces. This switches the dispatch model so that environments are inherently aware of how each dependency work, rather than dependencies adjusting themselves for each environment. In effect, the internal structure of the code should be simplified.
- Use os.TempDir() as the default workdir.
- Fix formatting of help output.
- Switch internal.TestContext interface to be a struct to reduce code clutter.

* [test-framework] Implement Mixer related infrastructure for local environment. (istio#5805)

* Add local-Mixer infrastructure and implement a basic test to cover.

* Accommodate CR feedback.

* Separate out the settings.

* [test-framework] Adding local pilot (istio#6077)

* [test-framework] Add cluster support to the test framework. (istio#6179)

* Add cluster support to the test framework.

* CR feedback.

* Minor linter fixes.

* [test-framework] Integrating local Pilot and Envoy (istio#6332)

- Updated configuration of Envoy to use discovery v2 API on Pilot

- Adding a discovery proxy to allow interception of responses from
Pilot. This will provide the ability to modify the Envoy configs to
support local testing.

- Updated the agent test to incorporate a local pilot. Also starting
to hash out how to determine whether or not an Envoy has received
a configuration update.

* [test-framework] Refactoring pilot test application (istio#6489)

Simplifying the structure to make the code more readable and to make it
better fit in with the application model of the new local agent.

* [test-framework] A few fixes for local envoy code (istio#6537)

* [test-framework] More robust port reservation (istio#6736)

This PR reserves a block of ports, so that future attempts to pick
a port will not collide with previous attempts.

* [test-framework] Add control RPC to test service (istio#6737)

The RPC allows us to simulate in-mesh traffic from one service to
another.

Creating a copy of the pilot test service under pkg/test. This is a
more appropriate home for the code and also avoids changing the rest
of the code that depends on the old service.

* [test-framework] Support local in-mesh traffic (istio#6743)

There are a lot of changes here that are interrelated.  Here's the summary.

1) Adds concept of a protocol client, which is used to intercept outbound
URLs. The interception is performed by the proxy which modifies the
URL host:port so that an outbound call from service A to B will first go
through service A's Envoy proxy.

2) Moved the discovery response filtering logic into the application proxy,
since the additional logic requires state from the proxy.

3) Updated the discovery filtering logic to intercept the outbound listeners
created by Pilot to enable communication with other services
(e.g. so A can talk to B). Pilot uses virtual listeners which are not
actually bound to a port (since outbound traffic would typically be
redirected to 15001 in a cluster configuration). The updated logic
modifies these listeners by assigning a newly reserved port and forcing
the listener to bind to that port. The new port is then stored in a
port map, which is used to modify the URLs on outbound traffic from the
application.

* [test-framework] Remove tagging/label support. (istio#6975)

* Remove tagging/label support.

* remove ignore

* [test-framework] Add API Server support. (istio#7276)

* [test-framework] APIServer + Galley tests (istio#7277)

* [test-framework] Code cleanup: Move and simplify the core driver code. (istio#7286)

* Refactor the core driver code:

+ Created test/framework/ as the namespace of the internal/implementation
of the test framework.
+ Moved the top-level driver code to test/framework/driver
+ Simplified the code at the pkg/test level, leaving only framework.go
for top-level test framework operations.
+ Simplified code in framework.go to simply redirect to the driver.

* More code move:

+ Created framework/environments and moveed cluster & local there.
+ Renamed cluster to kubernetes.
+ Moved test/internal to test/framework/internal.
+ Moved test/tmpl to test/framework/tml.

Also renamed environment.Interface to framework.Environment.

* More shuffling:

+ Moved test/dependency =? test/framework/dependency.
+ Moved framework/environment.go => framework/environment/environment.go
This is needed to break the dependency cycle.
+ Moved environment variable flags, args code to framework/settings.
+ Moved test/framework(formerly operations.go) to test/api.go.
+ Moved driver code from framework/driver/... to framework/

* Fix the mutex error.

* Rename construct => new

* [test-framework] Support non-HTTP ports in local agent (istio#7419)

Also doing some other cleanup and build fixes.

* [test-framework] Componentize the test framework. (istio#7344)

* Refactor and componentize the test-framework.

* Accommodate CR feedback.

* [test-framework] Move pilot to components. (istio#7515)

* [test-framework] Some minor cleanup for pilot component (istio#7536)

* [test-framework] Adding support for component dependencies (istio#7598)

- Added Component interface, which is now implemented by all components

- Separate component registries for local and kubernetes

- Updated the dependency Tracker to initialize all dependencies of
a component before initializing the component, itself.

* [test-framework] Integrating test apps into framework (istio#7628)

* [test-framework] Updating jsonpb dependency

* [test-framework] fixing linter errors in generated files

Needed to rename the files so they'd be ignored.

* [test-framework] Removing t.Skip calls in unit tests.

* [test-framework] removing accidental edit
ostromart pushed a commit that referenced this pull request Aug 17, 2018
* Implement mTLS functionality in MCP.

Add utility code to load&watch certificates and build a TransportCredentials object.
Add an authenticaton check mechanism to admit/fail incoming streams.
Add a basic list-based implementation for authentication check.
Move test certificates from galley/validator folder to pkg/mcp/testing.

* Accommodate CR comments
#1
ostromart pushed a commit that referenced this pull request Jan 18, 2019
* Move Galley validation integration test to its own folder.

* Introduce Galley conversion integration tests.

- Implement a conversion test based on an input-output golden file
model that tests Galley's MCP output, based on K8s resource inputs in
YAML form.
- Make Galley Server restartable. This is done by making the event
registration restartable as well.
- Add a new local-only Galley component in the new test framework.
- Fix Makefile issue that causes test.integration.* targets to shadow
test.integration.*.kube targets.
- Introduce an in-memory mcp/client/Updater implementation for testing.

* Cleanup.

* make format

* Fix lint issues.

* Add missing close method.

* Try to appease Lint gods.

* One more sacrifice.

* Accommodate CR comments.

* Try to fix linter problem #1.

* linter fixes.

* "Fix" lint issue.

* "Fix" lint issue.
ostromart pushed a commit that referenced this pull request Mar 28, 2019
* Unblock ip6 tables to allow ipv6 traffic

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* Enable ip6tables when hostname return ipv6 address

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fix syntax error in istio-iptables.sh

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fix logic error in istio-iptables.sh

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fix another logic error

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* debug #1

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* Reverting debug related changes

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>
ostromart pushed a commit that referenced this pull request Aug 12, 2019
* Fixing iptabes ranges

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fix shellcheck errors

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fixing ci failures #1

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fixing ci failures #2

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* fixing ci failures #3

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>

* Addressing comments

Signed-off-by: Serguei Bezverkhi <sbezverk@cisco.com>
ostromart pushed a commit that referenced this pull request Jan 14, 2020
Adapt istio-installer after testing
ostromart added a commit that referenced this pull request Jan 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.