IndexCache, a new sparse attention optimizer, delivers 1.82x faster inference on long-context AI models

Processing 200,000 tokens through a large language model is expensive and slow: the longer the context, the faster the costs skyrocket. Researchers from Tsinghua University and Z.ai have built a technique called IndexCache that reduces up to 75% of redundant…













