why is answer_relevancy always nan？

[LillyChen]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
why is answer_relevancy always nan？

** output**
{'answer_relevancy': nan, 'faithfulness': 0.2308}

Code Examples

def eval_run()
  answer_relevancy = AnswerRelevancy()
  faithfulness = Faithfulness()

  dataset = [{
        "query": query,  # 问题
        "retrieved_contexts": [context],  # 上下文
        "response": answer  # 答案
    }]

    # 将 evalsets 列表转换为 Dataset 对象
    ds_case = Dataset.from_list(dataset)
    # 获取结果  获取上下文
    # 开始评测
    column_map = {"user_input": "query", "retrieved_contexts": "retrieved_contexts", "response": "response"}
    metrics = [answer_relevancy, faithfulness]
    return self.eval(dataset, column_map, metrics)


def eval(self, dataset, column_map, metrics):
    result = evaluate(
        dataset=dataset,
        metrics=metrics,
        llm=self.llm,
        embeddings=self.embeddings,
        column_map=column_map,
    )
    print(result)
    print(result.scores)
    df = result.to_pandas()
    print(df.head())
    return result

[jjmachan]

hey @LillyChen which model are you using?

Do you have any tracing setup? what you can do is easily setup tracing and check if these are the reasons for errors

[LillyChen]

hey @LillyChen which model are you using?

Do you have any tracing setup? what you can do is easily setup tracing and check if these are the reasons for errors

     os.environ["OPENAI_API_KEY"] = "sk-"
    self.llm = LangchainLLMWrapper(ChatOpenAI(
        openai_api_key="sk-",
        openai_api_base="https://dashscope.aliyuncs.com/compatible-mode/v1",
        model_name="qwen-plus",
        temperature=0.7,
        top_p=0.8,
    ))
    os.environ['DASHSCOPE_API_KEY'] = 'sk-'
    self.embeddings = LangchainEmbeddingsWrapper(DashScopeEmbeddings())

[jjmachan]

got it - i could be the model you are using, can you check the traces?

have you setup any tracing tools?

[LillyChen]

got it - i could be the model you are using, can you check the traces?

have you setup any tracing tools?

What tracing tools should be used for tracking? Please recommend it. Thank you

[jjmachan]

https://docs.arize.com/phoenix is an OSS one available that you could try