Loading a fine tuned model

[vikashg]

Hello everyone,
I came across an issue that I had overlooked in the past. I wonder if anyone of you faced a similar thing and/or know an explanation

So, I am using a pretrained model and I am finetuning it and then saving it. Nothing crazy here.
However, when it comes to doing predictions using this model, I load it as follows

model = models.inception_v3(pretrained_flag=False)
model.aux_logits=False
model.to(device)
model.load_state_dict(torch.load(saved_model_filename))

If I do this, I get very bad predictions almost 50 percent accuracy

However, if I turn the pretrained_flag to True, my answers are correct. I wonder why that is. My understanding was that using the line model=models.inception_v3(pretrained_flag=False), I am getting the model architecture definition and whatever weights are contained are overwritten using model.load_state_dict(torch.load(saved_model_filename)) so it should not matter if I am using the pretrained flag to be 0 or 1.

Am I missing something here ? Please advice
Thanks soo much everyone.

[Nic-Ma]

Hi @vikashg ,

May I know what this models.inception_v3 comes from? I think it's not MONAI network. Maybe this is some tricky design in it?
@ericspod May share more experience on it?

Thanks in advance.

[vikashg]

Hey @Nic-Ma no this is the torchvision based inceptionv3 model not the monai model. Just thought will ask here if someone had a similar experience as i couldn't find any information online and might be relevant as we so use transfer learning from torchvision in monai too right

[Nic-Ma]

Hi @yiheng-wang-nv @ericspod , have you model experts seen this issue before?

Thanks in advance.

[yiheng-wang-nv]

Hi @vikashg , I'm not sure which place was wrong, but if I met the same issue, I may do some comparisons like: check if same input tensor can get same output tensor between model 1 (which is used to produce the weights) and model 2 (which is used to load the weights)

[ericspod]

It would be worth looking at the function code for inception_v3 to see if anything done there would make a difference to what you're doing. You might have to dig farther into the implementation details of the model as well. One thing to try is load a pretrain model, rerandomise the weights, then load your weights again, if the result is good then something else with the network's state is relevant.

Does your saved model state cover all the weights of the model or only the ones you've fine tuned? If the latter then loading a non-pretrained model would give you something with random weights with only some restored to what you saved when you do load_state_dict. It could also be that the state of model.training is different if loaded pretrained or not, so it's worth double checking that model.eval() was called.

[vikashg]

Thanks @ericspod I am looking into it and nothing seems to stand out on its own. I save the model using torch.save(model.state_dict(), PATH) as [recommended] (https://pytorch.org/tutorials/beginner/saving_loading_models.html). I think I need to look may be using this method I am only saving the changed weights and not all the states. I can confirm model.eval() was called.

Another thing I might have to check "if by default the earlier layers of a model are frozen and are not trained and only the last layer is trained". But I thought it should be explicitly stated in the Mednist tutorial. I think because the training and evaluation are happening in the same jupyter notebook and the model definition is not explicitly called during the inference cycle, this problem was overlooked.