Improve redis semantic cache implementation

[tylerhutcherson]

Improve Redis semantic caching

Relevant issues

• Fixes Server failing to start while setting for Redis Semantic Cache [Bug]:  #3056
• Fixes [Bug]: New schema format introduced; please update schema spec. (type=value_error) [redis-semantic] #3363
• Fixes [Feature]: Allow Redis Semantic caching with custom Embedding models #4001

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation

Changes

• Use latest version of RedisVL (0.3.2) and fix pydantic & schema version issues
• Support isolated/customized index names to allow for multitenancy
• Support arbitrary embedding model in index schema provided through litellm.embeddings module
• Use built-in SemanticCache extension for cleaner code and processing
• Add TTL support to redis semantic cache (OOTB feature of Redis rather than a tack-on)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 11, 2024 0:26am

[rbs333]

Generally LGTM had a question about the todo

[ishaan-jaff]

Is this missing async support @tylerhutcherson ?

If yes, can you add async support on this PR. Majority of users are trying to use this our llm gateway - which requires all async functions

[tylerhutcherson]

Is this missing async support @tylerhutcherson ?

If yes, can you add async support on this PR. Majority of users are trying to use this our llm gateway - which requires all async functions

Hey @ishaan-jaff @krrishdholakia see my comment here:
#5412 (comment)

Let me know what you think!

[ishaan-jaff]

@tylerhutcherson

Async Support is a hard requirement for LiteLLM. Majority of our users use async. If we merge this and users use this in prod their litellm service will go down. We have seen this before when a non-async function was used.

We're happy to wait on this until redisvl add async support

[tylerhutcherson]

@tylerhutcherson

Async Support is a hard requirement for LiteLLM. Majority of our users use async. If we merge this and users use this in prod their litellm service will go down. We have seen this before when a non-async function was used.

We're happy to wait on this until redisvl add async support

Understood -- thanks for the clarity. Will see what we can do, thanks!

[tylerhutcherson]

FYI redis/redis-vl-python#214 We are close to finalizing support.

[tylerhutcherson]

Now that https://github.com/redis/redis-vl-python 0.3.3 is out with async sem cache support.... I have updated this branch to reflect that. Thanks!

@ishaan-jaff @krrishdholakia

[ishaan-jaff]

looks good - could you send a screenshot of your tests passing for you locally ? Happy to merge after that

[ishaan-jaff]

looks good - could you send a screenshot of your tests passing for you locally ? Happy to merge after that @tylerhutcherson

[tylerhutcherson]

looks good - could you send a screenshot of your tests passing for you locally ? Happy to merge after that @tylerhutcherson

Sure thing -- screenshot below. A few caveats:

• There seems to be some issue with async tests here. The poetry env does not include pytest-asyncio or anything so the test module skips all async tests and throws warnings like this?

litellm/tests/test_caching.py: 20 warnings
  /Users/tyler.hutcherson/Library/Caches/pypoetry/virtualenvs/litellm-g3ANcoZD-py3.11/lib/python3.11/site-packages/_pytest/python.py:183: PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
  You need to install a suitable plugin for your async framework, for example:
    - anyio
    - pytest-asyncio
    - pytest-tornasync
    - pytest-trio
    - pytest-twisted
    warnings.warn(PytestUnhandledCoroutineWarning(msg.format(nodeid)))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

• Also I don't have Azure API access nor a dependency called diskcache for these tests... so there are 3 tests that fail (seen below). All of the Redis cases pass using poetry run pytest litellm/tests/test_caching.py.

@ishaan-jaff