Explore other LLMs evaluation metrics

[dcecchini]

We can find other evaluation metrics and approaches for LLMs. One source is this blogpost.

[dcecchini]

Anther source is this new tool: FLASK

[ArkajyotiChakraborty]

Hey @dcecchini I would like to contribute to this issue.

[dcecchini]

Hi @ArkajyotiChakraborty , good! Seems that some of those approaches may be easy to implement. Let me know how it goes.

[ArkajyotiChakraborty]

Yeah, let me check and start around tomorrow. Will keep you updated, are you considering any particular deadline for this?

[dcecchini]

I think the deadline will depend on your findings of what to implement and how complex it is. Let's analyze it and make a plan.

[alytarik]

@dcecchini
I added my findings in the exploration file in teams, but here is also the takeaways. I believe QG-QA would be nice to add and I opened an issue for it (#817). For others, Entailment score can be used in summarization tasks and we can make the score to use configurable (currently I think default is rouge and cant be changed)

Main takeaways:
BERTScore: Using pairwise cosine similarity between reference and result tokens to calculate prec, recall, f1 etc.
WER: Uses edit distance between candidate and reference text by counting the number of insertions, deletions, substitutions to transform.
MoverScore: Calculated by comparing the similarity of token movements between the source and target texts.
Entailment Score: Useful for ensuring faithfulness in text-grounded generation tasks like text summarization.
G-Eval: Ask the model directly to score between 0-5 for some aspects, generally biased
QG-QA: Use a model to generate questions given context: mrm8488/t5-base-finetuned-question-generation-ap
GitHub: https://github.com/orhonovich/q-squared