Skip to content

Improve ncnn memory estimation #2352

@JeremyRand

Description

@JeremyRand

Splitting out from #2070 since this deserves its own issue.

Quoting @theflyingzamboni:

Since you have so much RAM, I'm wondering if you'd be willing to run some tests. I've done some testing on NCNN VRAM estimation in the past (though only with ESRGAN), and I can already tell you that this isn't really correct, and neither is what we already have in place. I've only got 8GB VRAM, so I could never extend my tests far enough to gather fully complete data.

What I can say is that:

  1. Model size is not strongly correlative with VRAM usage. This is because a model can be made larger simply by performing more convolutions, which does not matter for VRAM usage because they are being done in sequence. The only thing it definitely correlates with is how much VRAM it takes to store the model itself.
  2. Individual weight sizes have a correlation with VRAM usage when running a model.
  3. Scale needs to be accounted for, which our estimation does not currently do. This estimation is based around a 4x scale, but an 8x model will blow past it.

I abandoned this back in the day because there seemed to be further factors I couldn't account for with the data I had, but maybe we can finally figure it out. Unfortunately, I seem to have deleted the set of different scale/nf/nb ESRGAN models I generated for these tests. If I can remember how I generated them all, I could send them to you.

I'm hoping that there may be some way to parse the ncnn parameter file in a way that reveals the memory usage, so that we don't need to use heuristics and black-box RE like described above; I haven't investigated this yet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions