Skip to content

Pilot segmentation violation with ports defined in both service and service entry #21508

@albertomardomingo

Description

@albertomardomingo

Bug description
When defining ports in service of type externalName and on service entry, we got a [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x184a204] in pilot, causing it to go into crashloopbackoff

2020-02-26T11:01:01.384701Z	debug	Multiple plugins setup inbound filter chains for listener 10.192.139.114:8080, FilterChainMatch may not work as intended!
2020-02-26T11:01:01.385233Z	debug	attached HTTP filter with 6 http_filter options to listener "10.192.139.114_8080" filter chain 0
2020-02-26T11:01:01.385336Z	debug	attached HTTP filter with 6 http_filter options to listener "10.192.139.114_8080" filter chain 1
2020-02-26T11:01:01.385500Z	debug	attached 2 network filters to listener "0.0.0.0_5432" filter chain 0
2020-02-26T11:01:01.385519Z	debug	attached 2 network filters to listener "0.0.0.0_5432" filter chain 1
2020-02-26T11:01:01.385531Z	debug	buildSidecarOutboundListeners: multiple filter chain listener 0.0.0.0_5432 with 2 chains
2020-02-26T11:01:01.385652Z	debug	attached 2 network filters to listener "0.0.0.0_5432" filter chain 0
2020-02-26T11:01:01.385669Z	debug	attached 2 network filters to listener "0.0.0.0_5432" filter chain 1
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x184a204]

goroutine 329 [running]:
istio.io/istio/pilot/pkg/networking/core/v1alpha3.isFallthroughFilterChain(...)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener.go:2247
istio.io/istio/pilot/pkg/networking/core/v1alpha3.mergeTCPFilterChains(0xc0013efba0, 0x2, 0x2, 0xc0013ec930, 0xc000eef2e0, 0xc, 0xc000cf7308, 0xc000770240, 0x1, 0x1, ...)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener.go:2083 +0x634
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).buildSidecarOutboundListenerForPortOrUDS(0xc00013fa00, 0xc000770240, 0xc0000b0d00, 0xc000770240, 0xc000b03948, 0x1, 0x1, 0xc000b03920, 0x1, 0x1, ...)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener.go:1427 +0x1475
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).buildSidecarOutboundListeners(0xc00013fa00, 0xc0000b0d00, 0xc000770240, 0xc000645320, 0xc001180cb0, 0x1, 0x1)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener.go:909 +0x96f
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ListenerBuilder).buildSidecarOutboundListeners(...)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener_builder.go:201
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).buildSidecarListeners(0xc00013fa00, 0xc0000b0d00, 0xc000770240, 0xc000645320, 0xc000cf7650, 0xc00066d300)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener.go:304 +0xc6
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).BuildListeners(0xc00013fa00, 0xc0000b0d00, 0xc000770240, 0xc000645320, 0xc000cf7810, 0x4407d6, 0xc000b91140)
	istio.io/istio@/pilot/pkg/networking/core/v1alpha3/listener.go:284 +0x16e
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).generateRawListeners(0xc0001526c0, 0xc0009c5680, 0xc000645320, 0xc0013ce4c0, 0x1fff760, 0xc000e27bc8)
	istio.io/istio@/pilot/pkg/proxy/envoy/v2/lds.go:49 +0x6f
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushLds(0xc0001526c0, 0xc0009c5680, 0xc000645320, 0xc000627ec0, 0x16, 0x2, 0x0)
	istio.io/istio@/pilot/pkg/proxy/envoy/v2/lds.go:29 +0x7c
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).StreamAggregatedResources(0xc0001526c0, 0x227ee80, 0xc000e66b80, 0x0, 0x0)
	istio.io/istio@/pilot/pkg/proxy/envoy/v2/ads.go:270 +0xbfb
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._AggregatedDiscoveryService_StreamAggregatedResources_Handler(0x1ef7b80, 0xc0001526c0, 0x22730c0, 0xc0010e6240, 0x3398ee0, 0xc000b62800)
	github.com/envoyproxy/go-control-plane@v0.9.1-0.20191002184426-9d865299d2ff/envoy/service/discovery/v2/ads.pb.go:181 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000549600, 0x2283ec0, 0xc0009c5380, 0xc000b62800, 0xc00052c9f0, 0x3357ce0, 0xc0018b9f80, 0x0, 0x0)
	google.golang.org/grpc@v1.24.0/server.go:1199 +0xb2e
google.golang.org/grpc.(*Server).handleStream(0xc000549600, 0x2283ec0, 0xc0009c5380, 0xc000b62800, 0xc0018b9f80)
	google.golang.org/grpc@v1.24.0/server.go:1279 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0010aac50, 0xc000549600, 0x2283ec0, 0xc0009c5380, 0xc000b62800)
	google.golang.org/grpc@v1.24.0/server.go:710 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.24.0/server.go:708 +0xa1

Expected behavior
While I understand that it may not be a correct configuration, I would not expect a segmentation violation for a conflict

Steps to reproduce the bug

Appliying the yaml found in the attached zip test-crash.zip
causes pilot to crash once the nginx pod starts.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

client version: 1.4.3
citadel version: 1.4.5
egressgateway version: 1.4.5
egressgateway version: 1.4.5
galley version: 1.4.5
ilbgateway version: 1.4.5
ilbgateway version: 1.4.5
ingressgateway version: 1.4.5
ingressgateway version: 1.4.5
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
pilot version: 1.4.5
pilot version: 1.4.5
pilot version: 1.4.5
pilot version: 1.4.5
policy version: 1.4.5
policy version: 1.4.5
sidecar-injector version: 1.4.5
telemetry version: 1.4.5
telemetry version: 1.4.5
data plane version: 1.4.5 (9 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:42:56Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/386"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.9-gke.9", GitCommit:"a9973cbb2722793e2ea08d20880633ca61d3e669", GitTreeState:"clean", BuildDate:"2020-02-07T22:35:02Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

$ helm version
version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.7"}

In a different cluster with istio 1.4.3 we do not reproduce this crash.

How was Istio installed?

Using the helm charts in version 2, and migrated to helm 3.

Environment where bug was observed (cloud vendor, OS, etc)
Google Cloud Platform with a GKE cluster, but installed manually using helm.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions