You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current behavior of the llama.cpp LLM pipeline is to always set n_ctx=0. When n_ctx=0, the context size defaults to n_ctx_train which can be very large with some models.
This change will fallback to the default n_ctx when n_ctx=0 fails due to being out of memory. It will also allow n_ctx as a input parameter. If a manually set n_ctx is too large, this will fail since it's user-specified.