Open Issues Need Help
View All on GitHubLMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: The `save_decode_cache` configuration variable, which is intended to control whether the decode KV cache is saved, is currently not effective in `lmcache/vllm v1`. The task involves modifying `lmcache/integration/vllm/vllm_v1_adapter.py` to properly implement this optional saving functionality based on the config.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: This GitHub issue describes a bug in LMCache where, even after successfully loading KV cache for hit tokens, it redundantly attempts to store those same tokens again. This leads to unnecessary write operations to the cache, as observed when sending identical requests and seeing logs indicating all tokens are being stored instead of just the newly generated ones.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: Debug a peer-to-peer (P2P) key-value (KV) cache sharing issue in LMCache, an LLM serving engine. The problem is that a second instance of the engine cannot load caches stored on the disk of the first instance, even though the first instance successfully stored and retrieved the caches. The task involves analyzing the provided Python code and debugging the P2P mechanism to ensure proper cache loading across instances.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: Optimize the CI/CD pipeline for the LMCache project. This involves improving the handling of unit test failures (stopping the pipeline on failure), configuring appropriate concurrency for unit tests (single GPU) and end-to-end tests (two GPUs), and potentially correcting concurrency settings for unit and integration tests.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: Write documentation for AMD GPU support in LMCache, mirroring the existing vLLM AMD ROCm documentation. This involves detailing how to install and utilize AMD GPUs with LMCache.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: Add a configuration parameter to LMCache to allow users to specify the Infinistore link type (Ethernet or Infiniband) at runtime, instead of hardcoding it to Ethernet. This will improve flexibility and allow users to leverage Infiniband connections if available.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
AI Summary: Update the configuration files (lmcache-prefiller-config.yaml and lmcache-decoder-config.yaml) within the LMCache project's examples (1p1d and xp1d) to replace instances of `nixl_peer_host` and `nixl_peer_port` with `nixl_receiver_host` and `nixl_receiver_port`, respectively. This involves correcting the naming convention for the host and port used in the disagg_prefill configuration.
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer