Support for validation set instead of evaluating on test set directly.

[mahmoudyusuf94]

I see you only evaluate the test set in each epoch, can we add a validation set, with early stopping criteria based on the results/loss on this validation set?
this would also require a way to checkpoint the whole model in order to save the best model configuration against the dev set to be used against the test set at the end of training.

Please let me know if we can add that.
1- dev set support with early stopping criteria
2- checkpointing logic, to save and load the model.

One last question: Can you provide a way to train only the base model (BERT-based) without the GAN components, so that I take these numbers as a reference. So I can tell that the BERT-based model only got the following results against these results. And when we added GAN, we got these results.

[mahmoudyusuf94]

@crux82 maybe I can add the validation set logic if you're not interested.
But it'd be very helpful if you added a configuration or a way to enable only training the BERT-based model without GAN (as a reference) or even share another notebook for that (The last question above). I need this to be using the same components and configurations.
Thanks in advance.

[crux82]

Dear Mahmoud

I am sorry but what you asked is just to implement a "standard" BERT-based model.
I think that the web is full of examples of this kind of training.

As an example, I would suggest you take a look at a LAB material I prepared at:

https://github.com/crux82/AILC-lectures2021-lab

Unfortunately, I think that adding what you ask would just make the GAN-BERT example ... less clear.

Hope the above example is clear and useful to implement your baseline.

Bests

Danilo

[hoangthangta]

I see you only evaluate the test set in each epoch, can we add a validation set, with early stopping criteria based on the results/loss on this validation set? this would also require a way to checkpoint the whole model in order to save the best model configuration against the dev set to be used against the test set at the end of training.

Please let me know if we can add that. 1- dev set support with early stopping criteria 2- checkpointing logic, to save and load the model.

One last question: Can you provide a way to train only the base model (BERT-based) without the GAN components, so that I take these numbers as a reference. So I can tell that the BERT-based model only got the following results against these results. And when we added GAN, we got these results.

I think you just need to replace val_set by test_set and add a "if" condition to stop when meeting some defined criteria (ex. accuracy), then save the model at that point.