What does workers_per_gpu value actually mean?

[Divelix]

I was experimenting with PointPillars training on my laptop (GPU: RTX 3050 ti with 4 GB VRAM, CPU: 11th Gen Intel(R) Core(TM) i5-11300H @ 3.10GHz), so I had to decrease samples_per_gpu to 2 to make model training fit in my VRAM. While samples_per_gpu value is clear to me - it is batch size, workers_per_gpu is unclear. I found explanation in this issue that it stands for CPU cores. So, should I expect faster training with larger workers_per_gpu value? From this assumption I tried to compare time of computing first epoch for different workers_per_gpu values: 1, 2, 4, 8 (nproc says I have 8 cores). Results: training time is always around 11 minutes == no speedup. Help me understand this please and what value should I use?

[JingweiZhang12]

@Divelix, Hi, your understanding about worker_per_gpu is right.
Maybe the main time consumption of training is not in the data IO, so the training time has few fluctuations.
Also, the workers_per_gpu is not the only factor affecting the time of data IO.