Does this also improve the behavior at higher contexts? For me running Deepseek at higher contexts PP and TG both approach ~1 t/s.

For me running Deepseek at higher contexts PP and TG both approach ~1 t/s.

This indicates that your computer spends the entire time computing self attention for long enough context. If so, this PR will have zero impact on your long context performance.

This indicates that your computer spends the entire time computing self attention for long enough context.

I'm trying to understand but that explanation (at least to me) doesn't explain why at low context PP uses a lot more power than TG (as it is compute bound), but at higher context the power usage looks a lot closer to TG (which is memory/QPI bandwidth bound).

I don't have actual numbers (as I don't think the exact numbers matter) but the difference is stark enough for me to notice based on the CPU temperatures.

but at higher context the power usage looks a lot closer to TG (which is memory/QPI bandwidth bound).

Or is it rather the other way around (TG looks a lot closer to PP)? If you buy my explanation that for a large context all the time is spent in the self attention calculation, then there isn't that much of a difference between TG and PP: for DeepSeek each row in the KV cache multiples 128 rows of activations (K*Q and V*softmax(K*Q)), so the matrix multiplications in TG and PP have very similar characteristics (there isn't much of a difference between multiplying 128 rows and 128 x n_ubatch rows), and it is compute bound, not memory bound.

If you buy my explanation

I do, I was just trying to understand it.

Or is it rather the other way around (TG looks a lot closer to PP)? that for a large context all the time is spent in the self attention calculation, then there isn't that much of a difference between TG and PP: for DeepSeek each row in the KV cache multiples 128 rows of activations (K*Q and V*softmax(K*Q)), so the matrix multiplications in TG and PP have very similar characteristics (there isn't much of a difference between multiplying 128 rows and 128 x n_ubatch rows), and it is compute bound, not memory bound.

That makes sense.

I did attempt to look at the PCM data I had from earlier and just generated, and looked at CPU power usage and IPC but I'm not sure if the numbers are actually useful since I found during TG that it was causing paging (there really isn't much spare RAM on my system during inference).

Not a comprehensive test, but this PR531 does indeed speed-up PP as compared to main on my DeepSeek-R1-0528-IQ1_S.

So while not as dramatic given only 58 ffn_down_exps@iq1_m on this MoE, the iq1_s speed-ups are already merged into main so overall much faster than before.

The IQ1_S_R4 still benches faster for this specific configuration at least and seems to be the same speed on both this PR and main as I would expect.

Note, to keep it simple, I did not use -rtr to repack the attn/shexp tensors; so actual CPU-only scenario would likely be faster still.

DeepSeek-R1-0528-IQ1_S

• type f32: 361 tensors
• type q4_0: 61 tensors attn_k_b
• type iq1_s: 116 tensors ffn_(gate|up)_exps
• type iq1_m: 58 tensors ffn_down_exps
• type iq4_ks: 551 tensors everything else

DeepSeek-R1-0528-IQ1_S_R4

• type f32: 361 tensors
• type q4_0: 61 tensors attn_k_b
• type iq1_s_r4: 116 tensors ffn_(gate|up)_exps
• type iq1_m_r4: 58 tensors ffn_down_exps
• type iq4_ks: 551 tensors everything else

Importantly, llama-perplexity runs clean on PR531@72fd9faa so the new iq1_m implementation seems solid. Here's the values using -ctk f16:

• IQ1_S: Final estimate: PPL = 4.8910 +/- 0.02856
• IQ1_S_R4: Final estimate: PPL = 4.8805 +/- 0.02876 (computed back on PR494)

model=/mnt/raid/models/ubergarm/DeepSeek-R1-0528-GGUF/IQ1_S_R4/DeepSeek-R1-0528-IQ1_S_R4-00001-of-00003.gguf
#model=/mnt/raid/models/ubergarm/DeepSeek-R1-0528-GGUF/IQ1_S/DeepSeek-R1-0528-IQ1_S-00001-of-00003.gguf

numactl -N 0 -m 0 \
./build/bin/llama-sweep-bench \
    --model "$model" \
    -c 8704 \
    -ctk q8_0 \
    -mla 3 -fa \
    -fmoe \
    --no-mmap \
    --threads 80 \
    --threads-batch 128 \
    --numa numactl \
    --warmup-batch

The IQ1_S_R4 still benches faster for this specific configuration at least and seems to be the same speed on both this PR and main as I would expect.

This is because of the extremely high total_experts/active_experts=32 ratio in DeeSeek-V3. For u_batch size of 512 we are still far away from the regime where this new repacking scheme pays large dividends. Perhaps the gains will be bigger for u_batch = 1024 or even u_batch = 2048?

But yes, I see that this PR may not have the huge impact that it should because people have somehow decided that ik_llama.cpp is only good for very large MoE models, so they keep using llama.cpp for everything else, missing out big times on performance for CPU-only inference (and it isn't so that CPU performance is not discussed in the llama.cpp repository on a regular basis).

Perhaps the gains will be bigger for u_batch = 1024 or even u_batch = 2048?

Yes, looks like even with the high ratio of deepseek MoE, this new repacking scheme begins to outstrip the _r4 variants at high enough batch sizes on this CPU only test using same xeon 6980P as above:

PR531@72fd9faa DeepSeek-R1-0528-IQ1_S_R4 -b 4096 -ub 4096

PR531@72fd9faa DeepSeek-R1-0528-IQ1_S -b 4096 -ub 4096

missing out big times on performance for CPU-only inference

I might try quanting this qwen2.5-72b finetune moonshotai/Kimi-Dev-72B today. your recent improvements (and reading commit logs for ik/iqk_gemm on improving iq4/5_ks even more) will make 72B dense models much more usable for hybrid inferencing...

honestly, the biggest hurdle to general adoption of this fork, imo, is the lack of a pre-compiled distributible binary e.g. appimage format etc... my guess is the majority of possible end-users don't know how to apt-get install cuda-toolkit... i've been noodling on that challenge some at least for linux users, not sure on windows/macos...

Perhaps the gains will be bigger for u_batch = 1024 or even u_batch = 2048?

Yes, looks like even with the high ratio of deepseek MoE, this new repacking scheme begins to outstrip the _r4 variants at high enough batch sizes on this CPU only test using same xeon 6980P

I would be curious to the cutoff point. With something like ./bin/llama-bench [...] -p 32,64,128,256,512,1024,2048,3072,4096

I would be curious to the cutoff point. With something like ./bin/llama-bench [...] -p 32,64,128,256,512,1024,2048,3072,4096

It is model and quantization type dependent. But I'm not removing the _R4/_R8 quants, so everyone is free to do their performance evaluation and decide if to use this or go with the row-interleaved variant. For sure this is a big gain for people who don't want to get involved with repacking and all that stuff, but just want to run a mainline llama.cpp model they downloaded from HF or elsewhere. This also removes the need to worry about if the row-interleaved variant is supported on CUDA or not in case of hybrid inference.

For me running Deepseek at higher contexts PP and TG both approach ~1 t/s.

I had been so used to V3 where I never enabled high batch sizes with amb because I rarely requested over the default batch size of 512. But with R1 that is not in the case (due to thought tokens removal which results in reprocessing context).

I ran an experiment at high context, processing 4096 tokens (33640 to 37736) and this went from 2950 to 1619 seconds, and even a reduction in compute buffer (15387.76 MiB vs 9404.80 MiB).