🚀 The feature, motivation and pitch

Gemma-2 and new Ministral models use alternating sliding window and full attention layers to reduce the size of the KV cache.

The KV cache is a huge inference bottleneck and this technique could be fine-tuned into other models to make them much more memory efficient, especially for large batch sizes.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.