New-Deployment executor function is unable to initialize properly #2838
Description
hi, I'm currently trying to learn the fission New-Deployment executor, and I've been following the instructions at
https://fission.io/docs/usage/function/executor/ However, I've encountered an unexpected issue: After creating a function, the corresponding HPA, Service, and Deployment are not being initialized. When I use 'test,' it gives me the following error message: 'error':'Post "[http://127.0.0.1:8888/specialize](http://127.0.0.1:8888/specialize%5C)": dial tcp 127.0.0.1:8888: connect: connection refused.' This has left me quite puzzled. It's worth mentioning that when I use Poolmgr, everything works as expected.
I'm trying to learn fission's New-Deployment executor, but I've encountered an unexpected issue: After creating a function, the corresponding HPA, Service, and Deployment are not being initialized."
Expected result
I hope that Dome can run smoothly with the New-Deployment executor and auto-scale as needed.
Actual result
After creating a function, the corresponding HPA, Service, and Deployment are not being initialized."
To Reproduce
I've used the following installation command:
helm install --version v1.19.0 --namespace fission fission fission-charts/fission-all --set serviceType=NodePort --set routerServiceType=NodePort --set additionalFissionNamespaces[0]=default --set additionalFissionNamespaces[1]=pro --set additionalFissionNamespaces[2]=pre --set additionalFissionNamespaces[3]=test --set additionalFissionNamespaces[4]=dev
todo I'm not sure if the following errors will have an impact
Error: INSTALLATION FAILED: 40 errors occurred:
* roles.rbac.authorization.k8s.io "fission-buildermgr-fission-cr" already exists
* roles.rbac.authorization.k8s.io "fission-fluentbit" already exists
* roles.rbac.authorization.k8s.io "fission-executor-fission-cr" already exists
...
$ fission version
client:
fission/core:
BuildDate: "2023-05-18T12:33:09Z"
GitCommit: f955d118
Version: v1.19.0
server:
fission/core:
BuildDate: "2023-05-18T12:33:09Z"
GitCommit: f955d118
Version: v1.19.0
I'm using K3S because I believe it should be relatively simpler than K8S
$ k3s -v
k3s version v1.26.8+k3s1 (631bb3f0)
go version go1.20.7
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.8+k3s1", GitCommit:"631bb3f0feb6049ca4b2e71a3b55bcc45d0eed10", GitTreeState:"clean", BuildDate:"2023-09-05T19:59:36Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.8+k3s1", GitCommit:"631bb3f0feb6049ca4b2e71a3b55bcc45d0eed10", GitTreeState:"clean", BuildDate:"2023-09-05T19:59:36Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
$ fission env create --name nodejs --image fission/node-env -n dev --mincpu 100 --maxcpu 200 --minmemory 128 --maxmemory 256
poolsize setting default to 3
environment 'nodejs' created
$ fission fn create --name hello --env nodejs --code hello.js --executortype newdeploy --minmemory 64 --maxmemory 128 --minscale 1 --maxscale 6 --targetcpu 50 -n dev
Package 'hello-582d6c5f-7631-441f-836a-7c95aaa6843f' created
function 'hello' created
$ fission fn test --name hello -n dev
Error: Error calling function hello: 500; Please try again or fix the error: error sending request to function
===================================
Function: hello
Environment: nodejs
Namespace: dev
Pod: newdeploy-hello-dev-a370-f517a0d376df-6ffcdc98c6-l6q5w
Container: nodejs
Node: slave2
===================================
> fission-nodejs-runtime@0.1.0 start /usr/src/app
> node server.js
user code loaded in 0sec 0.265448ms
::ffff:127.0.0.1 - - [26/Sep/2023:08:02:31 +0000] "POST /specialize HTTP/1.1" 202 - "-" "Go-http-client/1.1"
===================================
Function: hello
Environment: nodejs
Namespace: dev
Pod: newdeploy-hello-dev-a370-f517a0d376df-6ffcdc98c6-l6q5w
Container: fetcher
Node: slave2
===================================
{"level":"info","ts":"2023-09-26T08:02:25.229Z","caller":"otel/provider.go:50","msg":"OTEL_EXPORTER_OTLP_ENDPOINT not set, skipping Opentelemtry tracing"}
{"level":"info","ts":"2023-09-26T08:02:25.229Z","caller":"app/server.go:119","msg":"fetcher ready to receive requests"}
{"level":"info","ts":"2023-09-26T08:02:25.229Z","caller":"httpserver/server.go:21","msg":"starting server","service":"fetcher","addr":":8000"}
{"level":"info","ts":"2023-09-26T08:02:25.237Z","logger":"fetcher","caller":"fetcher/fetcher.go:388","msg":"successfully placed","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd","location":"/userfunc/user"}
{"level":"info","ts":"2023-09-26T08:02:25.237Z","logger":"fetcher","caller":"fetcher/fetcher.go:722","msg":"calling environment v1 specialization endpoint","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd"}
{"level":"error","ts":"2023-09-26T08:02:25.237Z","logger":"fetcher","caller":"fetcher/fetcher.go:741","msg":"error connecting to function environment pod for specialization request, retrying","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd","error":"Post \"http://127.0.0.1:8888/specialize\": dial tcp 127.0.0.1:8888: connect: connection refused","stacktrace":"github.com/fission/fission/pkg/fetcher.(*Fetcher).SpecializePod\n\tpkg/fetcher/fetcher.go:741\ngithub.com/fission/fission/cmd/fetcher/app.Run.func1\n\tcmd/fetcher/app/server.go:90"}
{"level":"error","ts":"2023-09-26T08:02:26.238Z","logger":"fetcher","caller":"fetcher/fetcher.go:741","msg":"error connecting to function environment pod for specialization request, retrying","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd","error":"Post \"http://127.0.0.1:8888/specialize\": dial tcp 127.0.0.1:8888: connect: connection refused","stacktrace":"github.com/fission/fission/pkg/fetcher.(*Fetcher).SpecializePod\n\tpkg/fetcher/fetcher.go:741\ngithub.com/fission/fission/cmd/fetcher/app.Run.func1\n\tcmd/fetcher/app/server.go:90"}
{"level":"error","ts":"2023-09-26T08:02:28.238Z","logger":"fetcher","caller":"fetcher/fetcher.go:741","msg":"error connecting to function environment pod for specialization request, retrying","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd","error":"Post \"http://127.0.0.1:8888/specialize\": dial tcp 127.0.0.1:8888: connect: connection refused","stacktrace":"github.com/fission/fission/pkg/fetcher.(*Fetcher).SpecializePod\n\tpkg/fetcher/fetcher.go:741\ngithub.com/fission/fission/cmd/fetcher/app.Run.func1\n\tcmd/fetcher/app/server.go:90"}
{"level":"error","ts":"2023-09-26T08:02:31.239Z","logger":"fetcher","caller":"fetcher/fetcher.go:741","msg":"error connecting to function environment pod for specialization request, retrying","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd","error":"Post \"http://127.0.0.1:8888/specialize\": dial tcp 127.0.0.1:8888: connect: connection refused","stacktrace":"github.com/fission/fission/pkg/fetcher.(*Fetcher).SpecializePod\n\tpkg/fetcher/fetcher.go:741\ngithub.com/fission/fission/cmd/fetcher/app.Run.func1\n\tcmd/fetcher/app/server.go:90"}
{"level":"info","ts":"2023-09-26T08:02:31.259Z","logger":"fetcher","caller":"fetcher/fetcher.go:680","msg":"specialize request done","trace_id":"c5cb3fdc7aa80ca8a7bb7f6b8be96cdd","elapsed_time":6.029411151}
Options:
--name='' Function name
--method=[GET] HTTP Methods: GET,POST,PUT,DELETE,HEAD. To mention single method: --method GET
and for multiple methods --method GET --method POST. [DEPRECATED for 'fn create',
use 'route create' instead]
--header=[] (-H) Request headers
--body='' (-b) Request body
--query=[] (-q) Request query parameters: -q key1=value1 -q key2=value2
--timeout=1m0s (-t) Length of time to wait for the response. If set to zero or negative number, no
timeout is set
--dbtype='kubernetes' Log database type, e.g. influxdb (currently influxdb and kubernetes logs are
supported)
--subpath='' Sub Path to check if function internally supports routing
Global Options:
--server='' Server URL
--verbosity=1 (-v) CLI verbosity (0 is quiet, 1 is the default, 2 is verbose)
--kube-context='' Kubernetes context to be used for the execution of Fission commands
--namespace='' (-n) If present, the namespace scope for this CLI request
Usage:
fission function test [options]
Error: error getting function response
I tried manually creating an HPA, and I found that it's effective.
$ kubectl get hpa notejs-test-dome -n pre
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
notejs-test-dome Deployment/notejs-test-dome 0%/50% 1 10 1 36s
I'd like to test a function of type Poolmgr in the 'test' namespace.
fission fn test --name hello -n test
hello, world!
$ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system helm-install-traefik-crd-rcfjz 0/1 Completed 0 13d 10.42.0.4 master <none> <none>
kube-system helm-install-traefik-wbxzn 0/1 Completed 1 13d 10.42.0.3 master <none> <none>
kube-system svclb-traefik-7f3ed441-tpmzq 2/2 Running 4 (95m ago) 13d 10.42.0.21 master <none> <none>
kube-system traefik-57c84cf78d-5p4c7 1/1 Running 2 (95m ago) 13d 10.42.0.16 master <none> <none>
kube-system svclb-traefik-7f3ed441-2knml 2/2 Running 4 (95m ago) 13d 10.42.2.15 slave2 <none> <none>
cattle-fleet-system gitjob-85b85d5df8-566fx 1/1 Running 2 (95m ago) 13d 10.42.2.17 slave2 <none> <none>
kube-system svclb-traefik-7f3ed441-sgs9t 2/2 Running 2 (13d ago) 13d 10.42.1.13 slave1 <none> <none>
cattle-fleet-system fleet-controller-79bdbcd9b4-9bp7v 1/1 Running 2 (95m ago) 13d 10.42.2.18 slave2 <none> <none>
cattle-fleet-local-system fleet-agent-7687d78788-nvbfl 1/1 Running 1 (13d ago) 13d 10.42.1.14 slave1 <none> <none>
cattle-system rancher-webhook-788c48b988-4hpqt 1/1 Running 2 (95m ago) 13d 10.42.2.19 slave2 <none> <none>
kube-system coredns-59b4f5bbd5-fcjrx 1/1 Running 2 (95m ago) 13d 10.42.0.18 master <none> <none>
kube-system local-path-provisioner-76d776f6f9-rl2nw 1/1 Running 4 (70m ago) 13d 10.42.0.19 master <none> <none>
cattle-system rancher-5d7cd9bbd8-nt4vs 1/1 Running 2 (95m ago) 13d 10.42.0.20 master <none> <none>
kube-system metrics-server-68cf49699b-kfqlq 1/1 Running 4 (70m ago) 13d 10.42.0.17 master <none> <none>
cattle-system rancher-5d7cd9bbd8-nw2ch 1/1 Running 2 (95m ago) 13d 10.42.2.16 slave2 <none> <none>
cattle-system rancher-5d7cd9bbd8-vsnh9 1/1 Running 1 (13d ago) 13d 10.42.1.15 slave1 <none> <none>
cattle-system helm-operation-x2m7m 0/2 Completed 0 59m 10.42.1.21 slave1 <none> <none>
cattle-system helm-operation-n2xqt 0/2 Completed 0 50m 10.42.2.25 slave2 <none> <none>
cattle-system helm-operation-9svlc 0/2 Completed 0 46m 10.42.1.35 slave1 <none> <none>
cattle-system helm-operation-495hw 0/2 Completed 0 19m 10.42.2.48 slave2 <none> <none>
fission webhook-5db9d6dfbd-6bmrp 1/1 Running 0 18m 10.42.1.58 slave1 <none> <none>
fission router-667fd9f9bf-kq4zk 1/1 Running 0 18m 10.42.2.50 slave2 <none> <none>
fission kubewatcher-6cd59cc64d-vp58v 1/1 Running 0 18m 10.42.1.61 slave1 <none> <none>
fission timer-c877f9c66-chg2w 1/1 Running 0 18m 10.42.1.59 slave1 <none> <none>
fission buildermgr-7866657dc7-ztbs2 1/1 Running 0 18m 10.42.1.62 slave1 <none> <none>
fission mqtrigger-keda-6d7d877fbc-5rfsk 1/1 Running 0 18m 10.42.1.60 slave1 <none> <none>
fission executor-7b57867bf7-jkrhq 1/1 Running 0 18m 10.42.0.34 master <none> <none>
fission storagesvc-b4cc86bcd-krnk8 1/1 Running 0 18m 10.42.2.51 slave2 <none> <none>
dev poolmgr-nodejs-dev-26164-666f6f8bd-cpjzc 2/2 Running 0 15m 10.42.0.35 master <none> <none>
dev poolmgr-nodejs-dev-26164-666f6f8bd-pj76m 2/2 Running 0 15m 10.42.1.63 slave1 <none> <none>
dev poolmgr-nodejs-dev-26164-666f6f8bd-4vbl4 2/2 Running 0 15m 10.42.2.52 slave2 <none> <none>
test poolmgr-nodejs-test-26720-5767bc599b-bkctc 2/2 Running 0 13m 10.42.0.41 master <none> <none>
test poolmgr-nodejs-test-26720-5767bc599b-flgz6 2/2 Running 0 13m 10.42.1.64 slave1 <none> <none>
test poolmgr-nodejs-test-26720-5767bc599b-68mr2 2/2 Running 0 12m 10.42.2.55 slave2 <none> <none>
pre notejs-test-dome-5b94b7fbd5-5vfv7 1/1 Running 0 84s 10.42.0.43 master <none> <none>