@xiezhq-hermann Hi xiezhq , why do you think a three layer GPU / CPU / File based kv cache storage mechanism is better ? To my best understanding , remove cpu layer and direct GDS transfer between GPU and File would be faster

@xiezhq-hermann Hi xiezhq , why do you think a three layer GPU / CPU / File based kv cache storage mechanism is better ? To my best understanding , remove cpu layer and direct GDS transfer between GPU and File would be faster

Good point @didoteebin the rationale lies in different hardware characteristics, for most systems, the PCIe bandwidth could enable us to do layer-wise overlapping of KV cache transfer and forward computation, (i.e., concurrently execute forward of layer 1 and loading KV caches for layer 2) as PCIe-5 could achieve about 50gb/s and about 20gb/s for PCIe-4 and the latency is more predictable. However, for most local disks and remote storage, the latency would be much higher and throughput would be lower. As a result we implemented different policies across different layers. Specifically, layer-wise zero overhead overlapping between GPU and CPU memory, and best-effort prefetching for storage devices. But I sure can see different hardware platform impact the design choice, e.g., a fast RDMA remote memory pool could make direct access more benefitial and we will keep evolving our systems to fit more users' need.
Thanks again for your PR as well, would love to integrate direct access as an alternative in near future as we complete the integration of some popular backend like Mooncake and 3FS.

I tested the performance of the hierarchical cache with benchmark_multiturn.py, and found that its performance actually degraded somewhat compared to the original implementation. As I understand it, at the beginning of the program, using the hierarchical cache requires storing the KV Cache, which introduces additional overhead. Then, I continued profiling with nsys and noticed that the D2H bandwidth was too low in the logs. After checking the source code, I found that when attention_backend="fa3", it performs a direct copy rather than zero-copy. What is the rationale behind this design choice?

I tested the performance of the hierarchical cache with benchmark_multiturn.py, and found that its performance actually degraded somewhat compared to the original implementation. As I understand it, at the beginning of the program, using the hierarchical cache requires storing the KV Cache, which introduces additional overhead. Then, I continued profiling with nsys and noticed that the D2H bandwidth was too low in the logs. After checking the source code, I found that when attention_backend="fa3", it performs a direct copy rather than zero-copy. What is the rationale behind this design choice?

@soyail there is a bug associated with co-running fa3 backend and kv cache io loading kernel, which we are still investigating. For now we would recommend using flashinfer as the attention backend for hicache.

I tested the performance of the hierarchical cache with benchmark_multiturn.py, and found that its performance actually degraded somewhat compared to the original implementation. As I understand it, at the beginning of the program, using the hierarchical cache requires storing the KV Cache, which introduces additional overhead. Then, I continued profiling with nsys and noticed that the D2H bandwidth was too low in the logs. After checking the source code, I found that when attention_backend="fa3", it performs a direct copy rather than zero-copy. What is the rationale behind this design choice?

@soyail there is a bug associated with co-running fa3 backend and kv cache io loading kernel, which we are still investigating. For now we would recommend using flashinfer as the attention backend for hicache.

@xiezhq-hermann Is there any update on this issue?

Some questions need assistance in answering, thanks a lot
I've conducted benchmarks on this feature using the Deepseek R1 model with 16 H20 GPUs, where HiCacheFile is configured to use tmpfs. During the tests, I noticed that eviction performance might be causing watchdog timeouts. Are there any specific metrics or logging mechanisms that could help me analyze this issue in more detail? This set of test loads performs well in the version with only enable-hierarchical-cache enabled.
SGLang errorlog as below

params.SamplingParams object at 0x7f33ca0fe6e0>)], available_size=1160, evictable_size=361907,
2025-07-15 17:47:33 - ERROR - Pyspy failed to dump PID 566194. Error: /bin/dash: 1: py-spy: not found

2025-07-15 17:47:33 - ERROR - Watchdog timeout (self.watchdog_timeout=300)

FileSystem info

tmpfs           512G   36G  477G   7% /tmp/hicache

The startup script for one of the sglang nodes is as follows

GLOO_SOCKET_IFNAME=eth0 \
NCCL_SOCKET_IFNAME=eth0 \
NCCL_IB_GID_INDEX=3 \
NCCL_IB_HCA=mlx5_ \
NCCL_IB_DISABLE=0 \
NCCL_MIN_NCHANNELS=24 \
NCCL_IB_QPS_PER_CONNECTION=8 \
MODEL_LENGTH=131072  \
TORCHINDUCTOR_FX_GRAPH_CACHE=1 \
TORCHINDUCTOR_AUTOGRAD_CACHE=1 \
TORCHINDUCTOR_CACHE_DIR="/data00/torch_compile/" \
SGL_ENABLE_JIT_DEEPGEMM=1 \
SGL_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 \
EPMOE_USE_DEEPGEMM=1 \
python3 -m sglang.launch_server \
  --cuda-graph-bs 1 2 4 8 10 16 20 24 28 32 40 48 56 64 72 76 78 80 82 \
  --cuda-graph-max-bs 82 \
  --attention-backend fa3 \
  --speculative-algo NEXTN \
  --speculative-draft /data00/DeepSeek-R1-NextN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 2 \
  --speculative-num-draft-tokens 4 \
  --model-path /data00/DeepSeek-R1 \
  --tp 16 \
  --dist-init-addr 192.168.0.29:20000 \
  --nnodes 2 \
  --node-rank 0 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --enable-ep-moe \
  --max-running-requests 82  \
  --disable-chunked-prefix-cache \
  --enable-hierarchical-cache \
  --host 0.0.0.0 \
  --port 8080 \
  --hicache-storage-backend file

Hi @yapple, TP is not fully supported yet as there are still some designs about config details to be figured out. Will fix it ASAP : )

hello, has this problem been fixed?

I tested the performance of the hierarchical cache with benchmark_multiturn.py, and found that its performance actually degraded somewhat compared to the original implementation. As I understand it, at the beginning of the program, using the hierarchical cache requires storing the KV Cache, which introduces additional overhead. Then, I continued profiling with nsys and noticed that the D2H bandwidth was too low in the logs. After checking the source code, I found that when attention_backend="fa3", it performs a direct copy rather than zero-copy. What is the rationale behind this design choice?

@soyail there is a bug associated with co-running fa3 backend and kv cache io loading kernel, which we are still investigating. For now we would recommend using flashinfer as the attention backend for hicache.

@xiezhq-hermann Is there any update on this issue?

@Charles-L-Chen the latest main has adopted a new backend selection mechanism, by default it will use fa3 for prefill and flashinfer for decoding to avoid this problem.

Some questions need assistance in answering, thanks a lot
I've conducted benchmarks on this feature using the Deepseek R1 model with 16 H20 GPUs, where HiCacheFile is configured to use tmpfs. During the tests, I noticed that eviction performance might be causing watchdog timeouts. Are there any specific metrics or logging mechanisms that could help me analyze this issue in more detail? This set of test loads performs well in the version with only enable-hierarchical-cache enabled.
SGLang errorlog as below

params.SamplingParams object at 0x7f33ca0fe6e0>)], available_size=1160, evictable_size=361907,
2025-07-15 17:47:33 - ERROR - Pyspy failed to dump PID 566194. Error: /bin/dash: 1: py-spy: not found

2025-07-15 17:47:33 - ERROR - Watchdog timeout (self.watchdog_timeout=300)

FileSystem info

tmpfs           512G   36G  477G   7% /tmp/hicache

The startup script for one of the sglang nodes is as follows

GLOO_SOCKET_IFNAME=eth0 \
NCCL_SOCKET_IFNAME=eth0 \
NCCL_IB_GID_INDEX=3 \
NCCL_IB_HCA=mlx5_ \
NCCL_IB_DISABLE=0 \
NCCL_MIN_NCHANNELS=24 \
NCCL_IB_QPS_PER_CONNECTION=8 \
MODEL_LENGTH=131072  \
TORCHINDUCTOR_FX_GRAPH_CACHE=1 \
TORCHINDUCTOR_AUTOGRAD_CACHE=1 \
TORCHINDUCTOR_CACHE_DIR="/data00/torch_compile/" \
SGL_ENABLE_JIT_DEEPGEMM=1 \
SGL_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 \
EPMOE_USE_DEEPGEMM=1 \
python3 -m sglang.launch_server \
  --cuda-graph-bs 1 2 4 8 10 16 20 24 28 32 40 48 56 64 72 76 78 80 82 \
  --cuda-graph-max-bs 82 \
  --attention-backend fa3 \
  --speculative-algo NEXTN \
  --speculative-draft /data00/DeepSeek-R1-NextN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 2 \
  --speculative-num-draft-tokens 4 \
  --model-path /data00/DeepSeek-R1 \
  --tp 16 \
  --dist-init-addr 192.168.0.29:20000 \
  --nnodes 2 \
  --node-rank 0 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --enable-ep-moe \
  --max-running-requests 82  \
  --disable-chunked-prefix-cache \
  --enable-hierarchical-cache \
  --host 0.0.0.0 \
  --port 8080 \
  --hicache-storage-backend file

Hi @yapple, TP is not fully supported yet as there are still some designs about config details to be figured out. Will fix it ASAP : )

hello, has this problem been fixed?

@Alisehen the TP issues should have been resolved already in latest main for the file backend, would you mind take another try?

@xiezhq-hermann Do you have any plan to support hicache in decode node in PD disaggrecation?

@xiezhq-hermann Do you have any plan to support hicache in decode node in PD disaggrecation?

Yes, @ShangmingCai is pushing for a solution that fetching cache on P node and writing to cache pool on D node.