Failing to evaluate my RAG with = Azure (generate_testset) error :The API deployment for this resource does not exist . on a working deployment

[shayefarti]

I have a valid endpoint in Azure OpenAI with a working model that I use for ad applications.
Looking at the following instructions, as I understand, I need to set it up in order to work with Azure."

Azure OpenAI Client Setup

My application involves taking a Confluence page and creating an evaluation test for it using the generate_testset function. However, every time I run it, I encounter the same error. I tried using a different Python version, 3.9 or 3.11.

as far as I understand, this is the way to set it for a Azure

My Code :

import sys
print(sys.version)
import pandas as pd
import os
from langchain_community.document_loaders import ConfluenceLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=20)

loader = ConfluenceLoader(
    url='https:/******wiki.jira.com/wiki/',
    username='*****@****',
    api_key='<Confluence key >',,
    page_ids=['4309844478'],
    keep_newlines=True,
    keep_markdown_format=True,# Hypothetical parameter to enable OCR
    ocr_languages='eng',
    limit=150,
    max_pages=2000,# Hypothetical parameter to set OCR language to English
)
documents = loader.load_and_split(text_splitter)
df = pd.DataFrame([d.page_content for d in documents], columns=["text"])
from giskard.rag import generate_testset
os.environ['AZURE_OPENAI_API_KEY'] = '*****'
os.environ['AZURE_OPENAI_ENDPOINT'] = 'https://cieapi.openai.azure.com/'
os.environ['OPENAI_API_VERSION'] = '2023-07-01-preview'
from giskard.rag import KnowledgeBase
knowledge_base = KnowledgeBase(df)
for key, value in os.environ.items():
    print(f'{key}: {value}')
from giskard.llm import set_llm_model
set_llm_model('gpt-4')
testset = generate_testset(
    knowledge_base,
    num_questions=60,
    language='en',  # optional, we'll auto detect if not provided
    agent_description="A customer support chatbot for company X", # helps generating better questions
)

Output:

/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/bin/Python /Users/shayefrati/IdeaProjects/documentation-helper/rag_eval/generate_test.py 
3.9.6 (default, Feb  3 2024, 15:58:27) 

AZURE_OPENAI_API_KEY:*******
AZURE_OPENAI_ENDPOINT: https://cieapi.openai.azure.com/
OPENAI_API_VERSION: 2023-07-01-preview
2024-07-24 12:02:36,372 pid:23448 MainThread giskard.rag  INFO     Finding topics in the knowledge base.
Traceback (most recent call last):
  File "/Users/shayefrati/IdeaProjects/documentation-helper/rag_eval/generate_test.py", line 36, in <module>
    testset = generate_testset(
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/testset_generation.py", line 63, in generate_testset
    _ = knowledge_base.topics
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 240, in topics
    self._topics_inst = self._find_topics()
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 251, in _find_topics
    clustering = hdbscan.fit(self._reduced_embeddings)
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 208, in _reduced_embeddings
    self._reduced_embeddings_inst = reducer.fit_transform(self._embeddings)
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 192, in _embeddings
    self._embeddings_inst = np.array(self._embedding_model.embed([doc.content for doc in self._documents]))
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/llm/embeddings/openai.py", line 32, in embed
    response = self.client.embeddings.create(input=batch, model=self.model)
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/resources/embeddings.py", line 114, in create
    return self._post(
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1266, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/_base_client.py", line 942, in request
    return self._request(
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1046, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

Process finished with exit code 1

I've tried all different modules of Azure gpt4 gpt4o and 3.5 , but the issue persists. I'm certain that the endpoint and the API key work because I use them in other applications.

[henchaves]

Hello @shayefarti, thanks for reporting this.

Could you try to add the following code before calling the generate_testset method?

os.environ["GSK_LLM_API"] = "azure"

If it works, I'll update the docs to include this line.

[shayefarti]

Thanks for the quick response. I've added GSK_LLM_API as you suggested, but the issue still persists.
As far as my little experience in Azure I usually don't specify the model in Azure when I call LLM , but rather the deployment name that could be the case?
In any case, whether it's deployment or model, I use the same name (gpt-4 etc.. ).

Hey, just a quick question - when we call the LLM, do we use client.completions.create or client.chat.completions.create? I'm not very experienced with Python, so it would be helpful to see the function that calls LLM for debugging.
As far as I understand, Azure uses only client.chat.completions.create.

AZURE_OPENAI_API_KEY: **********
AZURE_OPENAI_ENDPOINT: https://cieapi.openai.azure.com/
OPENAI_API_VERSION: 2023-07-01-preview
GSK_LLM_API: azure

2024-07-24 13:21:13,452 pid:53099 MainThread giskard.rag  INFO     Finding topics in the knowledge base.
Traceback (most recent call last):
  File "/Users/shayefrati/IdeaProjects/documentation-helper/rag_eval/generate_test.py", line 37, in <module>
    testset = generate_testset(
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/testset_generation.py", line 63, in generate_testset
    _ = knowledge_base.topics
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 240, in topics
    self._topics_inst = self._find_topics()
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 251, in _find_topics
    clustering = hdbscan.fit(self._reduced_embeddings)
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 208, in _reduced_embeddings
    self._reduced_embeddings_inst = reducer.fit_transform(self._embeddings)
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/rag/knowledge_base.py", line 192, in _embeddings
    self._embeddings_inst = np.array(self._embedding_model.embed([doc.content for doc in self._documents]))
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/giskard/llm/embeddings/openai.py", line 32, in embed
    response = self.client.embeddings.create(input=batch, model=self.model)
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/resources/embeddings.py", line 114, in create
    return self._post(
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1266, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/_base_client.py", line 942, in request
    return self._request(
  File "/Users/shayefrati/IdeaProjects/cloud-monitoring/custom_prometheus_exporters/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1046, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

Process finished with exit code 1

[henchaves]

Hey @shayefarti. Thanks for testing it.

I think the problem is happening with the embedding model. Do you know if you have any embedding models deployed?
The default one that Giskard tries to use from Azure is text-embedding-ada-002, so if this one is deployed, it should be working fine.

Regarding the LLM call, it uses client.chat.completions.create method

[shayefarti]

Yes I have one named text-embedding-3-large
How can I make it use this instead of text-embedding-ada-002 ?

[henchaves]

You can add this code snippet before calling generate_testset

from openai import AzureOpenAI
from giskard.llm.embeddings.openai import OpenAIEmbedding
from giskard.llm.embeddings import set_default_embedding

set_default_embedding(OpenAIEmbedding(client=AzureOpenAI(), model="text-embedding-3-large"))

[shayefarti]

It works. Thank you very much.

[henchaves]

@shayefarti good to know it has worked!
I plan to add a new method called set_embedding_model(similar to set_llm_model), to make this process easier when using Azure/OpenAI without the default embed model (text-embedding-ada-002)