Azure storage account queue scaler should allow users to select peek or approximate method #3555
Description
Proposal
Azure storage account queue scaler supports users defining the "peek" or "approximate" methods for evaluating the queue length.
Use-Case
Using Azure storage account queues to trigger a mix of short and longer running jobs using the Azure Functions runtime.
Anything else?
I have encountered a problem where KEDA ignores invisible messages (messages currently being processed) when evaluating the queue length for an Azure storage account queue. This gives a few outcomes:
- Fewer pods are scheduled than should be as currently in progress messages are not considered
- KEDA scales in pods more aggressively than it should, resulting in in progress messages being interrupted (most commonly encountered with longer running jobs)
A few issues have been raised in the past:
Scaled jobs have been suggested instead for long running tasks, but this doesn't necessarily solve the issues of:
- Excluding in progress messages from the queue count impacts short running jobs as well, it's just more noticeable on longer running jobs
- Users using the Azure Functions runtime to manage trigger lifecycle and per node scaling. To the best of my knowledge this runtime does not offer "execute one message" processing as required for scaled jobs
Based on the discussion in Issue 2385, it would be good if a user has the option to select the peek method, or approximate method.
As noted on Issue 2385, when using Azure functions on an App Service Plan, you can scale based on storage queue length. The implementation Azure has used for this is approximate message count, and does not offer any other method of scaling based on queue length.
A screenshot from the app service plan scale rules GUI: