Is there an existing issue for the same feature request?

• I have checked the existing issues.

Is your feature request related to a problem?

Memory index(live index) may consume huge amount of memory. Now memory index dump happens when row count encounters a fixed threshould.

Describe the feature you'd like

We should statistic the memory consumption of each memory index. And determine which index to dump accordingly when quota is exceeded.

Describe implementation you've considered

1. Add a memory index tracer in infinity_context. It contains an atomic variable as global memory index mem usage. The tracer also contains a container of BaseMemoryIndex instance pointer.
2. Add memory usage field in BaseMemoryIndex. When insert, add_doc or such operation happens, the information in BaseMemoryIndex should be updated. The global tracer should also be notified how much memory has been used for that new insertion (add a global atomic variable). The latter will check the cur mem usage and quota and submit a dump signal when condition meets.
3. The dump signal will awake a background task. The dump task will determined which index to dump according to the information of live index container in tracer.

Documentation, adoption, use case

No response

Additional information

No response