v0.1.5

What's Changed

• fix: handle string score in node filter by @shahules786 in #755
• Add ipython notebook demonstrating the ragas-haystack integration. by @annthurium in #754
• Update Bring Your Own LLM Docs Link by @pmbaumgartner in #765
• fix: edge case in answer_correctness by @shahules786 in #778
• Added metadata field to datarows by @rnbokade in #782
• fix: is_async not passed for llm generation in context recall by @jokokojote in #777
• fix: Nan in metrics/test set gen by @shahules786 in #786
• Fix adapt issue for keyphrase extractor by @smodlich in #766
• updated context_scoring_prompt and provide extra examples. by @nofrish in #725
• fix: faithfulness by @shahules786 in #787

New Contributors

• @annthurium made their first contribution in #754
• @rnbokade made their first contribution in #782
• @smodlich made their first contribution in #766
• @nofrish made their first contribution in #725

Full Changelog: v0.1.4...v0.1.5

v0.1.4

What's Changed

• docs: Add docs for new entity based metric by @sky-2002 in #674
• Fix typos that cause fatal assertion errors in prompts by @joy13975 in #676
• fix: edge case in faithfulness by @shahules786 in #682
• docs: Fix the precision calculation in context-precision metric doc by @amit-timalsina in #685
• fix: improving reproducability of evals when running everything together by @jjmachan in #671
• added octlane docs by @jjmachan in #695
• Docs: Fix content precision description by @jokokojote in #693
• Docs: Created own section for Calculation (instead of Dropdown) by @GeroVanMi in #696
• Fix: Enable Critique Prompt Customization in AspectCritique by @aaronkim218 in #700
• chore: improve bug/question template by @shahules786 in #702
• docs: clean and smooth by @shahules786 in #703
• Docs: Fix and supplement answer relevancy description by @jokokojote in #705
• fix: remove unwanted quotes from questions by @shahules786 in #708
• Readme Formatting by @gkamradt in #710
• docs: customising embeddings and llms by @shahules786 in #712
• chore: add copyright by @shahules786 in #716
• feat: test set generation improvements by @shahules786 in #719
• Adding Tonic Validate integration notebook by @akamor in #707
• fix: edge case in CR by @shahules786 in #728
• fix(ragas): remove mutable defaults by @mspronesti in #684
• fix: remove experimental filters by @shahules786 in #743
• Correct Faithfulness Verdict Map by @pmbaumgartner in #739
• feat(generator): extend construction to any langchain LLM and Embeddings by @mspronesti in #670
• docs: minor corrections by @shahules786 in #747

New Contributors

• @amit-timalsina made their first contribution in #685
• @jokokojote made their first contribution in #693
• @GeroVanMi made their first contribution in #696
• @aaronkim218 made their first contribution in #700
• @gkamradt made their first contribution in #710
• @akamor made their first contribution in #707
• @mspronesti made their first contribution in #684
• @pmbaumgartner made their first contribution in #739

Full Changelog: v0.1.3...v0.1.4

v0.1.3

What's Changed

• feat: improved node selection and question fixing by @shahules786 in #644
• docs: add explanation to metrics by @shahules786 in #658
• fix: edge cases in metrics by @shahules786 in #661
• Update evolution by @cuent in #666
• feat: Add new "entity-based context recall" metric by @sky-2002 in #667
• fix: filter and context selection by @shahules786 in #672
• Add support for optional max concurrency by @joy13975 in #643

New Contributors

• @cuent made their first contribution in #666
• @sky-2002 made their first contribution in #667

Full Changelog: v0.1.2...v0.1.3

v0.1.2

What's Changed

• docs: arize/ ragas notebook by @axiomofjoy in #594
• fix: prompt adaptation type checks by @shahules786 in #615
• fix: node indexing in testset generator by @shahules786 in #617
• Update ragas-arize.ipynb by @PubliusAu in #618
• docs: getting started by @jjmachan in #589
• Added testset generation for bedrock by @VpkPrasanna in #626
• Made embeddings and LLMs dependent on metric in evaluate function by @peanutshawny in #628
• docs: bring your own llms and embeddings by @jjmachan in #631
• docs: fix import in the example code: ContextRelevance -> ContextRelevancy by @izikeros in #646
• Fix typo: Teset -> Testset by @joy13975 in #638
• Update ragas-arize.ipynb by @PubliusAu in #637
• renaming ground_truths to ground_truth. by @alexjyong in #640
• fix: added types so you can pass langchain llms and embeddings too by @jjmachan in #652

New Contributors

• @axiomofjoy made their first contribution in #594
• @PubliusAu made their first contribution in #618
• @VpkPrasanna made their first contribution in #626
• @peanutshawny made their first contribution in #628
• @izikeros made their first contribution in #646
• @joy13975 made their first contribution in #638
• @alexjyong made their first contribution in #640

Full Changelog: v0.1.1...v0.1.2

v0.1.1

What's Changed

• Fixing typos, vertextai to vertex ai by @mrchtr in #572
• fix typo by @Kyeongpil in #570
• fix assertions and quality in prompt adaptation by @shahules786 in #574
• fix: ensure dict type by @shahules786 in #578
• style: remove additional comments in prompt.py by @leehanchung in #577
• Add stepkurniawan community contribution by @stepkurniawan in #585
• Parse JSON string in adapt() by @mitmul in #584
• fix: replace string formating by dict by @shahules786 in #588
• [patch] Reduce NaN Occurrences by Simple Prompt Modification for JSON Output for context_precision by @i-w-a in #581
• ci: migrate to latest llama-index version by @shahules786 in #603
• Fix: evolution flows for test generation by @shahules786 in #602
• fix: llamaIndex latest version by @jjmachan in #604
• Chore: add language to event tracking by @shahules786 in #592
• feat(llms.json_load): Recursively load json lists by @pberger514 in #593

New Contributors

• @mrchtr made their first contribution in #572
• @Kyeongpil made their first contribution in #570
• @leehanchung made their first contribution in #577
• @stepkurniawan made their first contribution in #585
• @mitmul made their first contribution in #584
• @i-w-a made their first contribution in #581
• @pberger514 made their first contribution in #593

Full Changelog: v0.1.0...v0.1.1

v0.1.0

What's Changed

• add initial embeddings.encode_kwargs by @lucasiscovici in #371
• feat(testset_generator): support error catching in generation process by @yuukidach in #368
• added class for FastEmbed by @tinomaxthayil in #379
• Updated NLI Verdicts to Binary Format by @lucasiscovici in #384
• added docs for ragas embeddings by @tinomaxthayil in #386
• fix context in _answer_correctness.py by @lucasiscovici in #385
• docs: AI copilot implementation by @nickscamara in #294
• fix: faithfulness for evasive/reject answers by @shahules786 in #392
• Added prompt class by @tinomaxthayil in #388
• Context precision with ground truth by @shahules786 in #391
• Added note to answer_relevance by @shahules786 in #399
• Import evaluate function before patching RagasLLM, fixes #301 by @HySoaKa in #377
• Fix: Added ground_truths arg type-hint info in the eval function by @MANISH007700 in #404
• Typo and phrasing correction by @TruscaPetre in #408
• Small improvements to documentation custom LLMs by @TruscaPetre in #410
• feat(metrics): skip embedding process when the corresponding weight is zero by @yuukidach in #409
• added prompt objects to metrics by @tinomaxthayil in #401
• feat: Automatic Prompt adaptation [language] by @shahules786 in #407
• Update _faithfulness.py by @Robs1999 in #416
• fix(metrics): avoid dividing by zero in correctness by @yuukidach in #421
• Feature/doc update ragas score by @prufrock349 in #417
• Docs: Prompts and prompt adaptation by @shahules786 in #419
• fix(metrics): wrong usage of prompt by @yuukidach in #422
• fix: context_utilization name override by @jjmachan in #424
• feat/executor: make ragas faster and more robust to run by @jjmachan in #390
• fix: Import PrompValue by @shahules786 in #427
• fix: add uuid to analytics by @shahules786 in #428
• feat: added async version for json_load by @jjmachan in #430
• fix: comment error in analytics by @jjmachan in #433
• fix: payload in analytics by @shahules786 in #434
• fix: tests for analytics by @jjmachan in #435
• Docs: tracing and logging by @shahules786 in #436
• feat: global prompt adaptation by @shahules786 in #432
• chore: remove mac artifacts from store by @jjmachan in #438
• fix: vary temperature based on strictness by @shahules786 in #437
• docs: custom llms by @shahules786 in #442
• feat: improving testset generation by @jjmachan in #381
• Fix: context_recall AttributeError by @0ENZO in #453
• Update faithfulness.md by @francesco-defalco in #451
• Fixing typo in faithfulness by @jamesbraza in #450
• fix: context_recall example by @shahules786 in #458
• fix: answer relevancy parsing by @shahules786 in #461
• feat: MVP for the new TestsetGenerator - SimpleEvolution by @jjmachan in #464
• fix: clean up embeddings for ragas and add docs for azure embeddings by @jjmachan in #477
• fix: polishing raise_exception and max_workers by @jjmachan in #486
• Convert prompts in testset generation by @shahules786 in #484
• feat(testset generation): implement new TestsetGenerator by @jjmachan in #487
• doc: add community page by @shahules786 in #488
• Fix ValueError: diag requires an array of at least two dimensions #474 by @herman159 in #475
• doc: modify prompt adaptation by @shahules786 in #490
• docs: theming community by @jjmachan in #491
• fix: CP example by @shahules786 in #489
• feat(testset): ported simple, multi_context and reasoning evolutions by @jjmachan in #496
• fix: cleanup old testset generator by @jjmachan in #500
• feat: added tracking of minimal stuff in testset generation by @jjmachan in #509
• feat: added generate_from_langchain function by @jjmachan in #511
• docs: updated testset generation by @shahules786 in #510
• fix: answer_correctness embedding by @jjmachan in #513
• added docs for customisation by @tinomaxthayil in #514
• fix: added rate_limter to langchain wrapper by @jjmachan in #517
• Update testset_generation.md by @Uzarel in #516
• fix: reshape matrix by @shahules786 in #519
• fix (metrics): changed ground_truths -> ground_truth by @jjmachan in #521
• docs: fix spelling by @shahules786 in #522
• fix: removed old prompt by @shahules786 in #523
• feat(testset): added conditional evolution by @jjmachan in #524
• fix: derive answer from relevant context by @shahules786 in #525
• feat(executor): remove the need to have both sync and async versions of the function by @jjmachan in #527
• feat: seed questions using keyphrases by @shahules786 in #526
• docs: updated contributors by @jjmachan in #532
• feat: configure retries and timeouts with evaluations and testset generators by @jjmachan in #534
• Fix case for few shots of context precision metric by @HuskyDanny in #541
• Updated imports, added dotenv by @aevedis in #539
• feat(executor): run executor in different thread to keep things clean by @jjmachan in #544
• add normalization factor to answer similarity score calculation by @enestydarealmc in #538
• Feat: prompt adaptation for test data generation by @shahules786 in #530
• fix:handle non dict output by @shahules786 in #547
• docs: adaptation for test set generator by @shahules786 in #548
• Fix: ensure quality of adapted prompts by @shahules786 in #546
• fix: raise warning for invalid JSON by @shahules786 in #550
• docs: adding code highlighting by @jjmachan in #552
• fix: wrap langchain embeddings automatically by @jjmachan in #553
• Added documentation page: Using Ragas with Athina for production performance metrics by @shivsak in #558
• fix: proper error message and fixed hf embedding error by @jjmachan in #561
• fix: answer_correctness doesn't reset stuff properly by @jjmachan in #562
• docs: fix issue template by @jjmachan in #565
• docs: fix issue template by @jjmachan in #566
• fix: handle non dict cases by @shahules786 in #564
• fix wrong type anno...

v0.1.0rc1

What's Changed

• add initial embeddings.encode_kwargs by @lucasiscovici in #371
• feat(testset_generator): support error catching in generation process by @yuukidach in #368
• added class for FastEmbed by @tinomaxthayil in #379
• Updated NLI Verdicts to Binary Format by @lucasiscovici in #384
• added docs for ragas embeddings by @tinomaxthayil in #386
• fix context in _answer_correctness.py by @lucasiscovici in #385
• docs: AI copilot implementation by @nickscamara in #294
• fix: faithfulness for evasive/reject answers by @shahules786 in #392
• Added prompt class by @tinomaxthayil in #388
• Context precision with ground truth by @shahules786 in #391
• Added note to answer_relevance by @shahules786 in #399
• Import evaluate function before patching RagasLLM, fixes #301 by @HySoaKa in #377
• Fix: Added ground_truths arg type-hint info in the eval function by @MANISH007700 in #404
• Typo and phrasing correction by @TruscaPetre in #408
• Small improvements to documentation custom LLMs by @TruscaPetre in #410
• feat(metrics): skip embedding process when the corresponding weight is zero by @yuukidach in #409
• added prompt objects to metrics by @tinomaxthayil in #401
• feat: Automatic Prompt adaptation [language] by @shahules786 in #407
• Update _faithfulness.py by @Robs1999 in #416
• fix(metrics): avoid dividing by zero in correctness by @yuukidach in #421
• Feature/doc update ragas score by @prufrock349 in #417
• Docs: Prompts and prompt adaptation by @shahules786 in #419
• fix(metrics): wrong usage of prompt by @yuukidach in #422
• fix: context_utilization name override by @jjmachan in #424
• feat/executor: make ragas faster and more robust to run by @jjmachan in #390
• fix: Import PrompValue by @shahules786 in #427
• fix: add uuid to analytics by @shahules786 in #428
• feat: added async version for json_load by @jjmachan in #430
• fix: comment error in analytics by @jjmachan in #433
• fix: payload in analytics by @shahules786 in #434
• fix: tests for analytics by @jjmachan in #435
• Docs: tracing and logging by @shahules786 in #436
• feat: global prompt adaptation by @shahules786 in #432
• chore: remove mac artifacts from store by @jjmachan in #438
• fix: vary temperature based on strictness by @shahules786 in #437
• docs: custom llms by @shahules786 in #442
• feat: improving testset generation by @jjmachan in #381
• Fix: context_recall AttributeError by @0ENZO in #453
• Update faithfulness.md by @francesco-defalco in #451
• Fixing typo in faithfulness by @jamesbraza in #450
• fix: context_recall example by @shahules786 in #458
• fix: answer relevancy parsing by @shahules786 in #461
• feat: MVP for the new TestsetGenerator - SimpleEvolution by @jjmachan in #464
• fix: clean up embeddings for ragas and add docs for azure embeddings by @jjmachan in #477
• fix: polishing raise_exception and max_workers by @jjmachan in #486
• Convert prompts in testset generation by @shahules786 in #484
• feat(testset generation): implement new TestsetGenerator by @jjmachan in #487
• doc: add community page by @shahules786 in #488
• Fix ValueError: diag requires an array of at least two dimensions #474 by @herman159 in #475
• doc: modify prompt adaptation by @shahules786 in #490
• docs: theming community by @jjmachan in #491
• fix: CP example by @shahules786 in #489
• feat(testset): ported simple, multi_context and reasoning evolutions by @jjmachan in #496
• fix: cleanup old testset generator by @jjmachan in #500
• feat: added tracking of minimal stuff in testset generation by @jjmachan in #509
• feat: added generate_from_langchain function by @jjmachan in #511
• docs: updated testset generation by @shahules786 in #510
• fix: answer_correctness embedding by @jjmachan in #513

New Contributors

• @lucasiscovici made their first contribution in #371
• @nickscamara made their first contribution in #294
• @HySoaKa made their first contribution in #377
• @TruscaPetre made their first contribution in #408
• @Robs1999 made their first contribution in #416
• @prufrock349 made their first contribution in #417
• @0ENZO made their first contribution in #453
• @francesco-defalco made their first contribution in #451
• @jamesbraza made their first contribution in #450
• @herman159 made their first contribution in #475

Full Changelog: v0.0.22...v0.1.0rc1

v0.0.22

What's Changed

• docs: comparing llms using ragas evaluations by @tinomaxthayil in #314
• convert context precision to json format by @shahules786 in #322
• fix: faithfulness.md by @kaiyikang in #328
• fix: structure faithfulness output by @shahules786 in #333
• fix: Column remap fix for Dataset evaluation by @tinomaxthayil in #331
• Fix OpenAIEmbeddings reference error in answer_relevancy by @ajinkyavbhandare in #329
• Fix: minor improvements across metrics by @shahules786 in #337
• fix wrong number by @Yongtae723 in #338
• fix: faithfulness for unrelated answering by @shahules786 in #340
• fix: load response as json by @shahules786 in #341
• docs: remove answer_relevancy from llm-docs by @tinomaxthayil in #343
• fix: bedrock llm error when evaluating rag qa validate_api_key by @arm-diaz in #350
• Added integration with Amazon API Gateway + bug fix in _answer_relevance.py by @brunopistone in #327
• feat: improve quality of answer correctness by @shahules786 in #339
• docs: add zeno visualization integration by @cabreraalex in #359
• fix(metrics): avoid getting none in context_recall by @yuukidach in #355
• fix: improved answer relevancy by @shahules786 in #346
• fix(types): fix pyright type issues with latest pyright version by @jjmachan in #366
• Retry for failed json loading by @shahules786 in #364
• fix: handle edge cases in prompt processing by @shahules786 in #374

New Contributors

• @kaiyikang made their first contribution in #328
• @ajinkyavbhandare made their first contribution in #329
• @brunopistone made their first contribution in #327
• @cabreraalex made their first contribution in #359
• @yuukidach made their first contribution in #355

Full Changelog: v0.0.21...v0.0.22

v0.0.21

What's Changed

• docs: Import AnswerSimilarity by @nelsonauner in #296
• FIx: testset_generator.py by @ferdinandl007 in #291
• Case-insensitive and whitespace handling for context_precision. by @Pauldevillers in #297
• Fix/tiny refactor by @Yongtae723 in #305
• fix: broken link to metrics by @younes-io in #311
• Convert context_recall to json format by @shahules786 in #312
• fix: add retry logic for OpenAI and Azure OpenAI by @jjmachan in #315
• fix: openai env var load after init and before score also by @jjmachan in #316
• import llama_index gracefully, solves #318 by @ceferisbarov in #319
• Make testgenerator output compatible with evaluate by @tinomaxthayil in #302

New Contributors

• @nelsonauner made their first contribution in #296
• @ferdinandl007 made their first contribution in #291
• @Pauldevillers made their first contribution in #297
• @younes-io made their first contribution in #311
• @ceferisbarov made their first contribution in #319

Full Changelog: v0.0.20...v0.0.21

v0.0.20

What's Changed

• docs: update how LLMs are customized in Ragas metrics by @jjmachan in #219
• fix: Add RagasLLM for customized test set generation by @chankeith96 in #242
• docs: add import statement in testset generation by @jjmachan in #249
• expose metrics prompts by renaming metric modules by @grauvictor in #243
• fix: added support for google vertexAI by @jjmachan in #252
• Renamed module names and made changes in init to reflect the change by @tinomaxthayil in #244
• added hint in synthetcic data generation by @tinomaxthayil in #259
• feat: RagasEmbeddings by @shahules786 in #232
• Fix: type hints by @shahules786 in #263
• Update issue templates by @shahules786 in #265
• Update evaluation.md typo by @whitewum in #275
• removed ragas score from result object by @tinomaxthayil in #276
• Custom embeddings from HuggingFace (sentence transformer) do not pass the assertion in src/ragas/metrics/_answer_similarity.py by @alebondarenko in #277
• fix: set temperate to near zero value by @shahules786 in #267
• feat: add native support for OpenAI and AzureOpenAI by @jjmachan in #261
• feat: make ragas pip more lighter by @jjmachan in #283
• docs: amazon bedrock (#241) by @arm-diaz in #246
• docs: added vertext ai support by @jjmachan in #284
• chore: pin OpenAI to >1 by @jjmachan in #285

New Contributors

• @chankeith96 made their first contribution in #242
• @grauvictor made their first contribution in #243
• @tinomaxthayil made their first contribution in #244
• @whitewum made their first contribution in #275
• @alebondarenko made their first contribution in #277

Full Changelog: v0.0.19...v0.0.20