How to Analyze Results #135
-
|
Hello, I am enjoying this repository, however I seem to be having trouble being able to analyze specific results. For example, I can get the base results as per the tutorial, but I am unable to find how to access specific answers for questions. (i.e. the model's generation for question 143 of HumanEval, and whether it was correct or incorrect) I was wondering if a feature like this is found within the MultiPL-E repository. The Tutorial's Example Page provides information on generated prompts, however I cannot seem to find where to access the content within my clone. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
They are produced in .results.json.gz files. I recommend looking at one to see the format. |
Beta Was this translation helpful? Give feedback.
-
|
Alternatively, several model completions from runs done for BigCode are here: https://huggingface.co/datasets/bigcode/MultiPL-E-completions |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for this great information! Is there a way to access the overall results for this raw data? for example when running a tutorial it gives this infromation: I have found this beautiful page that gives the overall results. Is this page the total results found from the data you linked? |
Beta Was this translation helpful? Give feedback.
They are produced in .results.json.gz files. I recommend looking at one to see the format.