It's definitely not great to do full iteration over the BPF map, but at the same time I don't quite like the idea of using the non-BPF data sources for producing the metric, for one because doing so would be rather complicated and secondly it would be error prone. Do we feel like doing full key iteration every 5 minutes on few potentially large maps (1-100k keys) be acceptable or should other options be considered instead? Or should we do this say only once every 30 minutes?

Another alternative is to keep a count to the side in lbmaps.go that's initialized when the map is opened by iterating once and then updated by each successful operation.

Or we can update the metric from Prune() that also iterates over the maps.

I ended up with this approach as it's the most generic, simplest and least error prone of the options and my only worry is the potential cost of it.

Another potentially more efficient way to do this would be to do this on BPF side using a BPF program running on a timer and iterating over all maps of our BPF programs and writing counts to a hash map, from which user-space would push them to metrics. Not quite sure how complicated that would be.

This seems like a good use case for BPF iterators (Linux v5.8, https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_TRACING/#iterator), where the iterator itself is a BPF program that is run against each element that could then accumulate and send the final value across the userspace barrier just once.

If we're concerned about the runtime cost of this implementation, I think it would be OK to have a hidden flag to disable the counting so that impacted users could dig through and disable this if they want to make that trade off. Otherwise as long as we come up with a reasonable default implementation I'd be OK with this as-is. I do think BPF iterators would provide a better solution but I don't feel that this would be a requirement to make LBMap pressure metrics available (don't want to increase the scope of this PR too much).

where the iterator itself is a BPF program that is run against each element that could then accumulate and send the final value across the userspace barrier just once

And we also have bpf_map_sum_elem_count kfunc, so it's one kfunc call (which sums per-cpu value) per map. Right now the way to use it is to iterate through all maps. If a subset of maps is needed one can load an iterator + a hashmap of maps of interest to filter all. (If only a small number of maps out of the whole system is required, we might consider optimizing this by either following up on adding per-map sum_elem_cout kfunc (see this thread), or adding a pre-filtered map iterator type.)

where the iterator itself is a BPF program that is run against each element that could then accumulate and send the final value across the userspace barrier just once

And we also have bpf_map_sum_elem_count kfunc, so it's one kfunc call (which sums per-cpu value) per map. Right now the way to use it is to iterate through all maps. If a subset of maps is needed one can load an iterator + a hashmap of maps of interest to filter all. (If only a small number of maps out of the whole system is required, we might consider optimizing this by either following up on adding per-map sum_elem_cout kfunc (see this thread), or adding a pre-filtered map iterator type.)

Yeah this would be great! We'd get map pressure metrics for all of the BPF maps this way and not just for a somewhat random subset that we do now. Wonder what'd be a good fallback when we don't have this supported. I guess we could still do the key iteration from user-space in those cases but just do it less often.

Using the map_sum_elem_count seems promising: #37927

I would like to suggest that we would go with this initially and then following up with the map_sum_elem_count approach later (that impacts other sub-systems and requires consensus, fallback implementations etc.).

This way I can soon switch the default LB control-plane to the new one without missing the pressure metrics.

/test

/test

/test

/test

/test

/test

/test

/test