Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting 'early_stop: 0' does not disable it #84

Closed
Andry-Bal opened this issue Dec 4, 2020 · 2 comments
Closed

Setting 'early_stop: 0' does not disable it #84

Andry-Bal opened this issue Dec 4, 2020 · 2 comments
Labels
bug Something isn't working

Comments

@Andry-Bal
Copy link

In the README it is said about early stopping that 'This feature can be turned off by passing 0'. This is, however, will not work. self.early_stop will be set to 0 in:

self.early_stop = cfg_trainer.get('early_stop', inf)

and later during training this condition will be true if current epoch was worse than previous:
if not_improved_count > self.early_stop:

leading to the output:
Validation performance didn't improve for 0 epochs. Training stops.

@Andry-Bal Andry-Bal changed the title Setting 'early_stop: 0' does not disable it. Setting 'early_stop: 0' does not disable it Dec 4, 2020
@SunQpark
Copy link
Collaborator

Thank you for reporting this, @Andry-Bal.
I believe simply adding a conditional statement after line 34 will fix this.

self.mnt_best = inf if self.mnt_mode == 'min' else -inf
self.early_stop = cfg_trainer.get('early_stop', inf)
if self.early_stop <= 0:
    self.early_stop = inf

I'll make a PR after doing some test. (or I'll appreciate if you make one)

Sorry for late response on this and other issue(#79) you reported. That's because the version of template I'm currently using for most of my projects is on hydra_DDP branch instead of master branch.

@SunQpark SunQpark added the bug Something isn't working label Dec 10, 2020
@SunQpark SunQpark mentioned this issue Dec 14, 2020
@Andry-Bal
Copy link
Author

No problem! Thank you for working on this project, it's really helpful.

SunQpark added a commit that referenced this issue Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants