Completion functions... Good idea?

[CholoTook]

Hi, I'd like to create an eval based on extracting scientific data from a PDF. I /think/ it would be useful to have a completion function that takes in a PDF (url or path), and outputs the JSON formatted text of the PDF... Am I thinking about this correctly?

Often the data in question will be in a table, which are often 'borked' when the PDF is converted to text.

Cheers,

[CholoTook]

I think I'm confused... A completion function is just a ChatCompletion as described here: https://platform.openai.com/docs/guides/chat/introduction ... I think...

[qrdlgit]

I was working on the exact same problem. There's been a few around this area: https://github.com/search?q=repo%3Aopenai%2Fevals+tables&type=pullrequests

I think for something like that, you could probably just do a straight match / fuzzymatch.

However, do check out the modelgraded evals, very cool. You use a second prompt to evaluate the results of the first prompt. You can imagine all the sophisticated things you can do there. logic-fact will get you started, but there are some other good examples.

Also note that as of last we heard, they aren't merging any custom code evals. #520 #715

There's also the fact that you'd be lucky to get your PR reviewed, never mind merged. :p #873 But I haven't let that stop me!

[CholoTook]

Yeah, I did it with Match in the end. However, what I want to do is pass the whole HTML (e.g. more realistic). But not sure how to do that... I think it doesn't fit the scope of what eval is trying to do. Many thanks for the link to the other evals.

…

On Thu, 4 May 2023 at 18:38, qrdlgit ***@***.***> wrote: I was working on the exact same problem. There's been a few around this area: https://github.com/search?q=repo%3Aopenai%2Fevals+tables&type=pullrequests I think for something like that, you could probably just do a straight match / fuzzymatch. However, do check out the modelgraded evals, very cool. You use a second prompt to evaluate the results of the first prompt. You can imagine all the sophisticated things you can do there. logic-fact will get you started, but there are some other good examples. — Reply to this email directly, view it on GitHub <#916 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANKSZTXWPGLILHVMA4DMYX3XEPSPRANCNFSM6AAAAAAXV3DAMU> . You are receiving this because you authored the thread.Message ID: ***@***.***>