You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current tfrecord dataset loads 1 tfrecord at a time into memory.
The deepspeed distributed wrapper causes the dataset to do this once, for every sample, for every GPU.
Maybe it would be best to preprocess / prefetch n samples, write them to disk, then load the correct sample from disk at train time.
The text was updated successfully, but these errors were encountered:
The current tfrecord dataset loads 1 tfrecord at a time into memory.
The deepspeed distributed wrapper causes the dataset to do this once, for every sample, for every GPU.
Maybe it would be best to preprocess / prefetch n samples, write them to disk, then load the correct sample from disk at train time.
The text was updated successfully, but these errors were encountered: