handle concurrent requests #2737

pranoyk · 2023-03-09T10:48:55Z

Description

Keep a track of number of pods in specialization and make check the capacity is more then requested else fail the function

Terminology

Capacity - (specialized pod + pods in specialization state) * rpp - (total active requests + functions awaiting svc)
svcWaiting - number of functions waiting for a svc

Approach

During getSvcValue if svc does not exist increase svcWaiting counter and check if capacity is greater then 1 (In the case of first request capacity would be less then 1)
If capacity is greater then 1 then push req into the queue and await specialization to finish
once specialization is done check if req are awaiting in queue for this function svc. If yes then allocate the svc to functions upto its requests per pod limit
keep repeating the process until the limit is reached.

Fixes #1939

Test scenarios

RPP	Concurrency	Requests	Expected	Actual
60	5	300	Specialized Pods - 5 Requests Success - 300 Requests Fail - 0	Specialized Pods - 5 Requests Success - 300 Requests Fail - 0
60	5	300 concurrent requests for 60 sec	Specialized Pods - 5 Requests Success - ~18000 Requests Fail - 0	Specialized Pods - 5 Requests Success - 14464 Requests Fail - 0 No of requests made can vary due to the test tool used (ab in this case)
1	10	10	Specialized Pods - 10 Requests Success - 10 Requests Fail - 0	Specialized Pods - 10 Requests Success - 10 Requests Fail - 0
5	5	25	Specialized Pods - 5 Requests Success - 25 Requests Fail - 0	Specialized Pods - 5 Requests Success - 25 Requests Fail - 0
5	1	5	Specialized Pods - 1 Requests Success - 5 Requests Fail - 0	Specialized Pods - 1 Requests Success - 5 Requests Fail - 0
5	1	6	Specialized Pods - 1 Requests Success - 5 Requests Fail - 1	Specialized Pods - 1 Requests Success - 5 Requests Fail - ~1 (requests gets timed out) In case the time taken to run the function is not very long then all 6 requests may be success
1	5	6	Specialized Pods - 5 Requests Success - 5 Requests Fail - 1	Specialized Pods - 5 Requests Success - 5 Requests Fail - 1 (requests gets timed out) In case the time taken to run the function is not very long then all 6 requests may be success
5	5	10	Specialized Pods - =<5 Requests Success - 10 Requests Fail - 0	Specialized Pods - =<5 Requests Success - 10 Requests Fail - 0
10	1	20	Specialized Pods - 1 Requests Success - 10 Requests Fail - 10	Specialized Pods - 1 Requests Success - 10 Requests Fail - 10

codecov · 2023-03-15T09:38:07Z

Codecov Report

Merging #2737 (df6998b) into main (b622f13) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2737   +/-   ##
=======================================
  Coverage   20.53%   20.53%           
=======================================
  Files          65       65           
  Lines        7191     7191           
=======================================
  Hits         1477     1477           
  Misses       5602     5602           
  Partials      112      112

Flag	Coverage Δ
unittests	`20.53% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pkg/executor/api.go

pkg/executor/executortype/container/containermgr.go

pkg/executor/executortype/newdeploy/newdeploymgr.go

pkg/executor/executortype/executortype.go

pkg/executor/executortype/poolmgr/gp.go

pkg/executor/fscache/poolcache.go

pkg/executor/executortype/container/containermgr.go

pkg/apis/core/v1/types.go

pkg/executor/executortype/newdeploy/newdeploymgr.go

pkg/executor/fscache/poolcache.go

…ming requests

… svc

…s and specialized is only more than concurrency limit

…rrency and rpp

Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>

LyKos4 · 2024-05-19T16:20:59Z

When I set concurrency 1 or rpp 1 and try to run a function more than 1 pods are created for that run, why does this happen?
Same for yolo true

Doesn't yolo true of rpp 1 mean that a function run should be done by a new single pod, meaning 1 pod per function run?

pranoyk requested a review from sanketsudake March 9, 2023 10:48

sanketsudake requested changes Mar 20, 2023

View reviewed changes

pkg/executor/api.go Outdated Show resolved Hide resolved

pkg/executor/api.go Outdated Show resolved Hide resolved

pkg/executor/executortype/container/containermgr.go Outdated Show resolved Hide resolved

pkg/executor/executortype/newdeploy/newdeploymgr.go Outdated Show resolved Hide resolved

sanketsudake reviewed Mar 21, 2023

View reviewed changes

pkg/executor/executortype/executortype.go Outdated Show resolved Hide resolved

pkg/executor/executortype/poolmgr/gp.go Outdated Show resolved Hide resolved

sanketsudake reviewed Mar 21, 2023

View reviewed changes

pkg/executor/fscache/poolcache.go Outdated Show resolved Hide resolved

pkg/executor/fscache/poolcache.go Outdated Show resolved Hide resolved

sanketsudake requested changes Mar 21, 2023

View reviewed changes

sanketsudake reviewed Mar 23, 2023

View reviewed changes

pkg/executor/fscache/poolcache.go Outdated Show resolved Hide resolved

pkg/executor/fscache/poolcache.go Outdated Show resolved Hide resolved

This was referenced Mar 28, 2023

fix the invalid fn.spec.Concurrency #2736

Closed

Relaxed concurrency in poolmanager strategy #1939

Closed

pranoyk force-pushed the feature/optimize-pool-manager-strategy branch from 988a801 to 1632f09 Compare March 28, 2023 12:17

pranoyk marked this pull request as ready for review March 28, 2023 14:13

pranoyk added 19 commits March 30, 2023 17:55

add functionality to wait for specialization by keeping track of inco…

54a2e09

…ming requests

format executor package

2acb7bf

fix required capacity to specialize new pod condition

b6fb0ad

move handling concurrency logic into pool cache from executor

5e6c3f3

remove unused methods and structs

6c6b35c

implement queue in to store the svc wait

4698310

create a queue struct and its methods to handle concurrent inputs

0302f40

use newly created queue to store waiting for svc requests

f361f42

add waiting requests in queue and use them when a svc is ready

d382363

set function to request in queue if the context is still alive

4a835e3

remove concurrency approach to set svc for waiting requests

6cf18f9

update the active requests whenever requests from pool are assigned a…

7e23670

… svc

add doc to define why the conditions exist

5df3775

remove unwanted params in strcut and clean up code

cdc2b85

set error while getting svc value if sum of specialization in progres…

9bd73b7

…s and specialized is only more than concurrency limit

remove duplicate functions and unnecessary values in struct

36afe5f

close svc channel on set value and create constants for default concu…

f3836a1

…rrency and rpp

get next value in queue in case context is timed out for fetched value

549d0b1

remove specializationInProgress counter from pool cache

f70ca7d

pranoyk and others added 8 commits March 30, 2023 17:55

return in case the queue is empty wihle setting func to svc

3077c6a

test getSvcVaue and setSvcValue in poolcache

5b303ec

add unit tests for GetConcurrent and GetRequestsPerPod methods

99a05db

reorder imports

5c28d24

add fuzzy testing for getSVCValue and setSVCValue in poolcache

6b65539

restructure go mod file and update pool cache test cases

acb3cce

Add tests and bug fixes

3d9674b

Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>

refactor code and add test cases

a5b3a5c

pranoyk force-pushed the feature/optimize-pool-manager-strategy branch from 29308c6 to a5b3a5c Compare March 30, 2023 12:28

add svcWaiting check while setting svc value

df6998b

sanketsudake approved these changes Mar 30, 2023

View reviewed changes

sanketsudake merged commit 715ef82 into main Mar 30, 2023

saltbo mentioned this pull request May 5, 2023

Provide parameter to limit yolo function max pods number #2785

Open

sanketsudake deleted the feature/optimize-pool-manager-strategy branch May 17, 2023 11:49

scott-rc mentioned this pull request May 23, 2023

Don't send 429 when max concurrency is reached gadget-inc/fission#19

Merged

sanketsudake mentioned this pull request Nov 21, 2023

Use concurrency in poolmanager as per old behaviour #2876

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

handle concurrent requests #2737

handle concurrent requests #2737

Uh oh!

pranoyk commented Mar 9, 2023 •

edited

Loading

Uh oh!

codecov bot commented Mar 15, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LyKos4 commented May 19, 2024

Uh oh!

Uh oh!

handle concurrent requests #2737

handle concurrent requests #2737

Uh oh!

Conversation

pranoyk commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Terminology

Approach

Test scenarios

Uh oh!

codecov bot commented Mar 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LyKos4 commented May 19, 2024

Uh oh!

Uh oh!

pranoyk commented Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 15, 2023 •

edited

Loading