Improvements to the resnet cifar10 example #490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fabianp
force-pushed
the
resnet_cifar10
branch
2 times, most recently
from
b657aad
to
8faea3d
Compare
fabianp
force-pushed
the
resnet_cifar10
branch
from
053f918
to
cd002f8
Compare
fabianp
changed the title
1. Reaches 91% accuracy with a resnet18 model on CIFAR10, before we were only getting 88%. The main change that made this possible was to tweak the size of the convolutional kernel in the first layer, from 7x7 (default) to 3x3. This is documented in the code, which now allows easily to overwrite the default kernel size. 2. Runs in 50 epochs (vs 200 before), and only takes 13 min on free colab, vs 1 hour before. 3. Removed many unused variables, and cleaned up the code. 4. I'm now using a warm-up learning rate, which seems to be working great across different architectures and datasets.
fabianp
force-pushed
the
resnet_cifar10
branch
from
cd002f8
to
30d38e7
Compare
mblondel
approved these changes
Aug 28, 2023
| :id: onkVLRw7L3j4 | ||
| :id: P7Z3Vex8QuGz | ||
|
|
||
| ModuleDef = Any |
There was a problem hiding this comment.
What do we gain from this definition? It wasn't entirely clear to me while reading the code. Maybe add a quick comment explaining why you introduce this.
There was a problem hiding this comment.
TBH I'm doing it because flax does it that way (https://github.com/google/flax/blob/main/examples/imagenet/models.py). I guess conv: ModuleDef is more explicit (for a human) than conv: Any , although the parser will treat them equally?
There was a problem hiding this comment.
If that's the Flax way, fine with me!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reaches 91% accuracy with a resnet18 model on CIFAR10, before
we were only getting 88%. The main change that made this possible was
to tweak the size of the convolutional kernel in the first layer, from
7x7 (default) to 3x3. This is documented in the code, which now allows
easily to overwrite the default kernel size.
Runs in 50 epochs (vs 200 before), and only takes 13 min on free
colab, vs 1 hour before.
Removed many unused variables, and cleaned up the code.
I'm now using a warm-up learning rate, which seems to be working
great across different architectures and datasets.