Evaluation functions

[EdwinB12]

Very simple application of using an evaluation function in prompto.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[EdwinB12]

This would work better as a method in experiment. The user would run it outside of experiment.process()

[EdwinB12]

Restriction on passed function is it must take in a prompt dictionary and it must return a prompt dictionary

[EdwinB12]

Should support a list/tuple of functions. Don't support arguments. Encourage the user to use the prompt dictionary to parameterise.

[EdwinB12]

This has ended up being a very bare bones application of this and i'm not sure what value it actually adds over just running an evaluation function on the completed responses dictionary saved to disk after called .process().

[rchan26]

thanks @EdwinB12 - looks great! this will definitely be useful in practice since having to manually post-run an evaluation on responses is not ideal. in the future, this will be a CLI command like the judge one

will merge this and I will add documentation pages for this

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 52.41%. Comparing base (cf15ce4) to head (824734a).
Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
src/prompto/experiment.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #94       +/-   ##
===========================================
+ Coverage   35.67%   52.41%   +16.74%     
===========================================
  Files          38       38               
  Lines        1962     1984       +22     
===========================================
+ Hits          700     1040      +340     
+ Misses       1262      944      -318     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.