Saving Keys and Values Cache at lower precision #6
Description
Refer https://github.com/FMInference/FlexGen - they have explored storing cache at 4-bit quantization.
{{ message }}
Refer https://github.com/FMInference/FlexGen - they have explored storing cache at 4-bit quantization.