Training Iteration

[ZYJ-JMF]

HI, thanks for your impressive work.
I am trying training your code from the begining, so I was wondering how many GPUS you used and how long will it spends for training?

I am looking forward to your response.Thanks

[williamyang1991]

I used one GPU with 8GB memory.
I don't remember the exact training time. I think it should take several days but within one week.
If you uses GPUs with more memory like 32GB, maybe you can use a large batchsize to speed up the training.

[Chanarin]

thanks so much for your amazing work!
I train your model with my own data but it takes so long time with my 6GB of GPU. Unfortunately, the electricity cut off. Can we resume training? I can't find any model file in the save directory. Please guide me on how to can I resume training.

[williamyang1991]

@Chanarin Unforunately, the code only save the final file after training. So it cannot be resumed.
You can modify train.py to add the save code during the training process so that it can be resumed if the electricity cut off.
https://github.com/williamyang1991/TET-GAN/blob/bdfca141fc14c5917fd9be8d2bc23870f9ad3288/src/train.py#L89C4-L89C58