Prompt caching | Context caching

“Caching is particularly impactful for applications inherently involving prompt repetition:

• Retrieval-Augmented Generation (RAG) / Document Q&A: Large documents or context snippets are often fed repeatedly as part of the prompt while only the user's question changes. Caching the document context significantly speeds up Q&A over that document.

• Few-Shot Prompting: Providing multiple examples within the prompt (few-shot learning) improves model performance but increases prompt length. Caching these static examples avoids reprocessing them for every new query.

• Agentic Workflows: AI agents often rely on complex system prompts, detailed instructions, and a fixed set of tool definitions. Caching these constant elements accelerates task execution, especially in multi-step processes.

• Chatbots / Multi-Turn Conversations: While the conversation history grows, the initial system prompt and instructions often remain the same. Caching the system prompt, and potentially incrementally caching turns of the conversation, keeps the interaction snappy even as the context window fills.

• Code Assistants: Static code context, library documentation, or boilerplate instructions can be cached, allowing the assistant to focus computation on the user's specific coding query.”
  
  

https://apidog.com/blog/what-is-prompt-caching/