Clarification on Task 1 Zero-Shot Batch Size and Memory Usage

[HaniyehBarghi]

Hi,

I'm running Task 1 zero-shot evaluation on an NVIDIA A100 (80 GB VRAM) using the default batch_size=1024 in gena_lm.py and encountering CUDA OOM errors. Could you please clarify what batch sizes you used for this task? If 1024 was used, how did you avoid running into memory issues on similar hardware?

Thanks for your help!

[austintwang]

Hi!

We did run our evaluations on similar hardware using the parameters we specified in the code. However, we noticed that GENA-LM's tokenizer has a tendency to produce large variations in sequence lengths, resulting in significant fluctuations in memory usage from batch to batch.

I'd expect that decreasing batch size would mitigate this, but let us know if issues persist.

[HaniyehBarghi]

Hi again,

reducing the batch size does indeed prevent the OOM errors, but the evaluation procedure is very slow. The estimation time this specific task is about 16 days to finish on one A100.

Did you observe a similar runtime in your experiments, or did you apply any additional optimizations to speed things up?

[austintwang]

16 days does seem like a long time. What batch size are you using?

[HaniyehBarghi]

Sorry for my late response.

Unfortunately, it is too long. For the zero-shot, I reduced the batch size to 512 (instead of the default 1024) to avoid CUDA OOM.

For Task 1 fine-tuning, I'm using a batch size of 32 (I ran into CUDA OOM when trying 64 or 128). At this setting, each epoch takes approximately one day to complete.

Do you have any suggestions for improving runtime beyond just reducing batch size? Also, are the packages listed in environment.yml sufficient for optimal performance?

I was considering switching to pure float16 to allow larger batch sizes and potentially speed up training. Do you think this would help, or are there other recommended strategies?

Also, even for Task 1 probing, that I did not encounter to CUDA OOM, every 20 epochs requires one day. This way to finish the 150 epochs it will be 7 days. Is this expected?

Thanks again for your support!

[PanosAntoniadis]

Hey! I have exactly the same problem. I decreased the batch size but now evaluation takes many many days to complete. Is there any update?

[HaniyehBarghi]

Not from my side. For now, I gave up on this task.

[PanosAntoniadis]

Did you encounter similar problems with the other tasks?

[HaniyehBarghi]

I reproduced the results for task 3 and task 4 only for GENA-LM (BERT-Large) model, and it was more manageable. It was a while ago, so I do not remember the exact runtime, but definitely less than one or two days.