Sample code to configure using Google Gemini Pro

[khannan31]

Has anybody managed to get autollm working with google gemini-pro with lancedb? Can you please share code for how to configure the llm_model, embed_model, etc. parameters within query_engine = AutoQueryEngine.from_defaults(.....). Also unsure who to specify the vertex AI project name and location.

[fcakyon]

easy as this:

import os
from autollm import AutoQueryEngine

os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811" # Your Project ID
os.environ["VERTEXAI_LOCATION"] = "us-central1" # Your Location

llm_model = "gemini-pro"

AutoQueryEngine.from_defaults(
     documents='...',
     llm_model=llm_model,
)

[khannan31]

Thank you for your quick response. I seem to be still missing something, since the code is still defaulting to use OpenAI. Is there something else that needs to be specified to get litellm to use gemini-pro? Also, what environment variable to create to specify the Gemini Api key?
Section of the Error Message below:
'''
2024-01-07 12:50:22,757 - autollm - INFO - Adding documents to existing database at ./.lancedb.

Parsing nodes: 100%
1/1 [00:00<00:00, 4.99it/s]
Generating embeddings: 0%
0/46 [00:00<?, ?it/s]

2024-01-07 12:50:22,985 - autollm - INFO - Embedding Token Usage: 0

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

---

OpenAIError Traceback (most recent call last)
File ~\AppData\Roaming\Python\Python311\site-packages\litellm\llms\openai.py:423, in OpenAIChatCompletion.embedding(self, model, input, timeout, api_key, api_base, model_response, logging_obj, optional_params, client, aembedding)
422 if client is None:
--> 423 openai_client = OpenAI(api_key=api_key, base_url=api_base, http_client=litellm.client_session, timeout=timeout, max_retries=max_retries)
424 else:

File ~\anaconda3\envs\autollm\Lib\site-packages\openai_client.py:92, in OpenAI.init(self, api_key, organization, base_url, timeout, max_retries, default_headers, default_query, http_client, _strict_response_validation)
91 if api_key is None:
---> 92 raise OpenAIError(
93 "The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable"
94 )
95 self.api_key = api_key

OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

During handling of the above exception, another exception occurred:

OpenAIError Traceback (most recent call last)
File ~\AppData\Roaming\Python\Python311\site-packages\litellm\main.py:1897, in embedding(model, input, timeout, api_base, api_version, api_key, api_type, caching, user, custom_llm_provider, litellm_call_id, litellm_logging_obj, logger_fn, **kwargs)
1896 ## EMBEDDING CALL
-> 1897 response = openai_chat_completions.embedding(
1898 model=model,
1899 input=input,
1900 api_base=api_base,
1901 api_key=api_key,
1902 logging_obj=logging,
1903 timeout=timeout,
1904 model_response=EmbeddingResponse(),
1905 optional_params=optional_params,
1906 client=client,
1907 aembedding=aembedding,
1908 )
1909 elif model in litellm.cohere_embedding_models:

File ~\AppData\Roaming\Python\Python311\site-packages\litellm\llms\openai.py:446, in OpenAIChatCompletion.embedding(self, model, input, timeout, api_key, api_base, model_response, logging_obj, optional_params, client, aembedding)
445 import traceback
--> 446 raise OpenAIError(status_code=500, message=traceback.format_exc())

OpenAIError: Traceback (most recent call last):
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\litellm\llms\openai.py", line 423, in embedding
openai_client = OpenAI(api_key=api_key, base_url=api_base, http_client=litellm.client_session, timeout=timeout, max_retries=max_retries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\anaconda3\envs\autollm\Lib\site-packages\openai_client.py", line 92, in init
raise OpenAIError(
openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

During handling of the above exception, another exception occurred:
'''

[fcakyon]

@khannan31 gemini-pro is an LLM type. gemini-pro cannot generate embedding vectors. For embedding generation, autollm defaults to ada embedding model from openai.

If you don't want to use openai for embedding generation, you can use an embedding model from huggingface as detailed here: #193 (comment)

[khannan31]

Thank you...

[khannan31]

Gemini does have embeddings. Was able to run this below code to get it.
`

https://ai.google.dev/tutorials/python_quickstart#use_embeddings

!pip install -U google-generativeai

import google.generativeai as genai

genai.configure(api_key=api_key)
result = genai.embed_content(
model="models/embedding-001",
content=[
'What is the meaning of life?',
'How much wood would a woodchuck chuck?',
'How does the brain work?'],
task_type="retrieval_document",
title="Embedding of list of strings")

A list of inputs > A list of vectors output

for v in result['embedding']:
print(str(v)[:50], '... TRIMMED ...')

`

[khannan31]

[0.0040260437, 0.004124458, -0.014209415, -0.00183 ... TRIMMED ...
[-0.004049845, -0.0075574904, -0.0073463684, -0.03 ... TRIMMED ...
[0.025310587, -0.0080734305, -0.029902633, 0.01160 ... TRIMMED ...