Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EVAL_STANDALONE_END documentation missing #3443

Open
ewirbel opened this issue Jul 1, 2024 · 1 comment · Fixed by #3445
Open

EVAL_STANDALONE_END documentation missing #3443

ewirbel opened this issue Jul 1, 2024 · 1 comment · Fixed by #3445
Labels
bug Something isn't working

Comments

@ewirbel
Copy link

ewirbel commented Jul 1, 2024

Expected behavior

EVAL_STANDALONE_END event is missing from the pseudo code definition in the Composer event doc. It looks like the pseudo code is referring to another name for the even, AFTER_EVAL_ALL which does not seem to exist anymore.

Additional it is not very clear that EVAL_END is called after evaluation on one evaluation dataset, so called several times if there are several datasets.

https://docs.mosaicml.com/projects/composer/en/latest/api_reference/generated/composer.Event.html#composer.Event.EVAL_STANDALONE_END

@ewirbel ewirbel added the bug Something isn't working label Jul 1, 2024
@mvpatel2000
Copy link
Contributor

EVAL_STANDALONE_END event is missing from the pseudo code definition in the Composer event doc

Ah, this is because the doc focuses on the training loop, and this is only run if you call just trainer.eval(...), which we do not provide docstrings for. We'd love a community PR describing the eval loop events after the train loop if you're up for it!

It looks like the pseudo code is referring to another name for the even, AFTER_EVAL_ALL which does not seem to exist anymore.

Ah, it should be EVAL_AFTER_ALL. I'll fix the typo.

Additional it is not very clear that EVAL_END is called after evaluation on one evaluation dataset, so called several times if there are several datasets.

It should be described in the for loop over evaluators in docstring -- is there a better place you'd like a more helpful description?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants