Large Language Models (LLMs) face significant computational challenges when
processing long contexts due to the quadratic complexity of self-attention.
While soft context compression methods, which map input text to smaller latent
representations, have shown promise, their real-world adoption is limited.
Existing techniques typically compress the context as a single unit, which
leads to quadratic compression complexity and an inability to reuse
computations across queries with overlapping contexts. In this work, we
introduce CompLLM, a soft compression technique designed for practical
deployment. Instead of processing the context holistically, CompLLM divides it
into segments and compresses each one independently. This simple design choice
yields three critical properties: efficiency, as the compression step scales
linearly with the context length; scalability, enabling models trained on short
sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and
reusability, allowing compressed segments to be cached and reused across
different queries. Our experiments show that with a 2x compression rate, at
high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x
and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance
comparable to that obtained with the uncompressed context, and even surpasses
it on very long sequences, demonstrating its effectiveness and practical
utility.