Testset generation feedback

[tuan3w]

Hello,

I would like to open this issue to discuss tips and guidance related to test set generation. If you don't think this is the right place, feel free to close this issue.

Recently, I tested with Ragas 0.2, making some modifications, and generated tests for Vietnamese. I do very simple setup. I use PyPDFLoader to load pdf docs.

Here are some observations:

1. It generates questions for short, unattractive contexts. Perhaps some rule-based filters could help address this issue.

2. I noticed that short, simple questions were generated even for long contexts. Tweaking the prompt slightly might improve the quality of the questions.

3. I feel that the generated questions are too generic. Perhaps adding a global theme or topic could enhance the question generation process.
4. Ultimately, I believe that having the ability to use custom prompts to control the question generation would be very beneficial.

Thanks

[shahules786]

Hey @tuan3w
thanks for your feedback. Can you also share how many documents you used and also describe the nature of the data?
Some of these are documentation related issues, for example item 4 - you can already customise prompts using set_prompts and get_prompts method as in here but it's not documented for testset.
Others would need bit more polishing. Test generation v3 is very capable, but it surely needs more polishing to make it better.

Also I just created #1577 to track feedbacks for it. Feel free to engage in conversations, and post your feedbacks there.

[ableiweiss]

@shahules786 can you share an example of how this would be done for testset? I use set_prompts to modify the prompt for SpecificQuerySynthesizer and verify that it changes the instruction and examples, but when I generate it doesn't have an effect.