Time and Space Limits for Deep Learning on UoS HPCs

[TheLostLambda]

Hello! I'm looking to retrain / train a larger model inspired by https://github.com/nadavbra/protein_bert and was wondering:

1. Where should I store training data? It's >1TB but it seems most storage locations that large on the Sheffield HPCs are temporary storage only?
2. Could that data be persisted and could I reserve GPU usage for a few weeks to a month? The original model was trained for a month, but this was likely using a lower-powered GPU!

The GitHub for ProteinBERT seems to have some pretty nice instructions for the retraining, I'm just wondering if I'll run into any issues using that much disk space and that much GPU time!

Thanks a ton,
Brooks

[willfurnass]

I suggest the /fastdata (Lustre filesystem) areas as they don't have quotas and they're the most performant areas when working with larger files. Files in those areas will be deleted after 60 days - is that too little time?

Can the training process easily be paused/resumed? If so, that gives you more flexibility as to how the workload can be run: it could potentially be split across multiple jobs, possibly using free CPU/GPU cycles (see https://docs.hpc.shef.ac.uk/en/latest/hpc/scheduler/index.html#preemptable-jobs)

If you want to discuss your GPU resource needs in more detail then do get in touch via [email protected].