Abstract: The increasing adoption of large language models (LLMs) with extended context windows necessitates efficient Key-Value Cache (KVC) management to optimize inference performance. Inference ...

Abstract: Improving software-managed cache efficiency is an important issue for various modern applications. Although LRU (Least Recently Used) has been widely used as the default replacement policy ...