Skip to content

High memory usage on blacklist refresh #470

@bkupidura

Description

@bkupidura

Im using blocky with ~19 blacklists, every refreshPeriod i see high memory usage spike (~50MB -> ~150MB -> ~50MB).

I tries to run all my pods with memory limits, to avoid OOM on HV, and this memory spikes sometimes leads to blocky killed by K8s.

Can we introduce some method to limit how many links will be downloaded in same time?
Afaik currently (

for _, link := range links {
) we are just going thru all lists, and trying to fetch them in same time. With big number of lists/big number of entries in list we can hit this memory spike.
If we would be able to limit how many lists should be downloaded in same time it should help.

EDIT: I see that there are some download options (e.g downloadTimeout), can we just add download
Concurrency? :)

Screenshot 2022-03-28 at 16 09 08

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions