Skip to content

zstd decoder not re-using memory effectively between Reset() and Read() #666

@bboreham

Description

@bboreham

I've been stepping through some decompression code to try to figure out why zstd memory use is much higher than other compression types.

I see that Reset() uses the buffer d.current.b:

dst = d.current.b

Then in my case DecodeAll runs and the buffer grows to ~300KB.

Each time Read() is called, it advances the start of the buffer:

d.current.b = d.current.b[filled:]

After we read the whole thing, b is still pointing into a 300KB buffer, but towards the end.
When we re-use the Decoder, this buffer has len zero and cap ~35KB. Because 300KB is needed, a new block is allocated.

Could we have the Decoder hold a reference to the beginning of the buffer, so the whole size can be re-used?

Apologies if I'm mis-reading this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions