Skip to content

Commit

Permalink
docs: add langfuse notebook (#216)
Browse files Browse the repository at this point in the history
How to use langfuse dashboard with ragas metrics
  • Loading branch information
jjmachan authored Oct 30, 2023
1 parent c1ed36e commit d85f02b
Show file tree
Hide file tree
Showing 7 changed files with 737 additions and 34 deletions.
Binary file added docs/_static/list-score-traces-ragas.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/traces-score-ragas.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/getstarted/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ pip install ragas
```

Ragas also uses OpenAI for running some metrics so make sure you have your openai key ready and available in your environment

```python
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
```
## The Data
Expand Down
56 changes: 24 additions & 32 deletions docs/getstarted/monitoring.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,27 @@
(get-started-monitoring)=
# Monitoring

Maintaining the quality and performance of an LLM application in a production environment can be challenging. Ragas provides a solution through production quality monitoring, offering valuable insights into your application's performance. This is achieved by constructing custom, smaller, more cost-effective, and faster models.

[**Get Early Access**](https://calendly.com/shahules/30min)

```{admonition} **Faithfulness**
:class: note
This feature assists in identifying and quantifying instances of hallucinations.
```

```{admonition} **Bad retrieval**
:class: note
This feature helps identify and quantify poor context retrievals.
```

```{admonition} **Bad response**
:class: note
This feature helps in recognizing and quantifying evasive, harmful, or toxic responses.
```

```{admonition} **Bad format**
:class: note
This feature helps in detecting and quantifying responses with incorrect formatting.
```

```{admonition} **Custom use-case**
:class: hint
For monitoring other critical aspects that are specific to your use case. [Talk to founders](https://calendly.com/shahules/30min)
Maintaining the quality and performance of an LLM application in a production environment can be challenging. Ragas provides with basic building blocks that you can use for production quality monitoring, offering valuable insights into your application's performance. This is achieved by constructing custom, smaller, more cost-effective, and faster models.

:::{note}
This is feature is still in beta access. You can requests for
[**early access**](https://calendly.com/shahules/30min) to try it out.
:::

The Ragas metrics can also be used with other LLM observability tools like
[Langsmith](https://www.langchain.com/langsmith) and
[Langfuse](https://langfuse.com/) to get model-based feedback about various
aspects of you application like those mentioned below

:::{seealso}
[Langfuse Integration](../howtos/integrations/langfuse.ipynb) to see Ragas
monitoring in action within the Langfuse dashboard and how to set it up
:::

## Aspects to Monitor

1. Faithfulness: This feature assists in identifying and quantifying instances of hallucinations.
2. Bad retrieval: This feature helps identify and quantify poor context retrievals.
3. Bad response: This feature helps in recognizing and quantifying evasive, harmful, or toxic responses.
4. Bad format: This feature helps in detecting and quantifying responses with incorrect formatting.
5. Custom use-case: For monitoring other critical aspects that are specific to your use case. [Talk to founders](https://calendly.com/shahules/30min)
1 change: 1 addition & 0 deletions docs/howtos/integrations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ happy to look into it 🙂
llamaindex.ipynb
langchain.ipynb
langsmith.ipynb
langfuse.ipynb
:::
Loading

0 comments on commit d85f02b

Please sign in to comment.