[Bug] CPU Mem utilization grows with training, when Dataloader num_workers>0

[BradZhone]

Describe the bug

CPU memory utilization grows with training and finally cause OOM when num_workers of Dataloader greater than 0.
Especially when more datasets are used, this mem growth phenomenon becomes more obvious.

Environment

torch 2.3.0+cu121

Other information

No response