Prompt Summarization

Replace a long passage with a shorter paraphrase. Higher quality than truncation but adds latency from an extra model call.

How It Works

A dedicated compression model or the target model itself (in a pre-pass) reads the long content and produces a condensed version preserving the key facts, decisions, and state. The summary replaces the original content in the prompt.

What to Summarize

• conversation history
• long documents or reference material
• tool outputs (logs, search results, test output)
• code files when only the interface matters

Trade-offs

• Pros: preserves semantic content better than truncation; can be tuned for the domain.
• Cons: adds latency and cost for the summarization step; quality depends on the summarizer.

See Also

• Prompt Truncation — faster but lower-quality alternative.
• Rolling Summarization — summarization applied to conversation history.
• Compaction — summarization is the primary compaction technique.

Tools

• LLMLingua — prompt compression library using smaller models to compress prompts while preserving semantic meaning.
• LangChain Contextual Compression — retriever wrapper that compresses retrieved documents before sending to the LLM.