长上下文大语言模型推动了众多下游应用的发展，但也带来了计算和内存效率方面的重大挑战。为了应对这些挑战，围绕 KV 缓存的长上下文推理优化方法应运而生。然而，现有基准测试通常仅关注单请求场景，忽视了 KV 缓存在实际使用中的完整生命周期。

Our LLM API bill was growing 30% month-over-month. Traffic was increasing, but not that fast. When I analyzed our query logs, I found the real problem: Users ask the same questions in different ways. ...