Skip to main content

Automatic caching for claude - add support for top-level cache_control

Please add support for top-level cache_control for claude models. It enables automatic caching: https://platform.claude.com/docs/en/build-with-claude/prompt-caching

Venice currently allows to add the cache_control parameter inside content blocks for claude models to cache up to these breakpoints, but Venice does not support enabling cache for the tool results, so in the best case (last role: assistant content block is marked with cache_control, role: tool results follow) users pay for the tool result input twice.

Venice should accept cache_control at the message-level of role: tool messages to fix this, but the best fix would be to support the top-level cache_control for automatic caching, which is also much easier to implement for you and your users.

Log in to comment and vote

No comments yet

Be the first to share your thoughts.