Skip to content

Allow configuring the LiteLLM cache directory - #1390

Open
lindicaphxag-tech wants to merge 6 commits into
huggingface:mainfrom
lindicaphxag-tech:contrib/litellm-cache-config-806
Open

lindicaphxag-tech wants to merge 6 commits into
huggingface:mainfrom
lindicaphxag-tech:contrib/litellm-cache-config-806

Conversation

@lindicaphxag-tech

Copy link
Copy Markdown

Closes #806

LiteLLMClient currently creates a disk cache at module import time, so the cache directory cannot be selected from a model configuration.

This adds an optional litellm_cache_dir to LiteLLMModelConfig. When it is set, the client configures LiteLLM's disk cache with that directory; leaving it unset keeps the existing default behavior. The example config and focused unit coverage are included.

Validation:

  • Python compilation of the changed module and test file
  • Isolated config/cache initialization test with a stubbed LiteLLM cache

@lindicaphxag-tech

Copy link
Copy Markdown
Author

Following up on #806: the optional litellm_cache_dir keeps the existing default when omitted, and the PR includes focused cache-initialization coverage plus an example config. Could a maintainer review when available and flag any compatibility concerns with LiteLLM cache initialization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FT] Allow configuration of litellm cache location

1 participant