-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Training Issue #19
Comments
Hi @DevonPeroutky, Thank you for your interest in our work. Did you try to upgrade the transformers to the latest version? Please note that LLaMA-3 based trainings are only supported with "transformers==4.41+" which you can install as follows,
Let me know if it helps. Good Luck! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment
Issue
I'm seeing random and sudden loss spikes during training, if there is a simpler way of debugging this, I'm open to a new approach. However, I attempted to reproduce the training loop in pytorch such that I could log out abnormal gradients during the training process to detect any erroneous examples in my training data.
However, I'm always getting
AttributeError: 'NoneType' object has no attribute 'device'
in the forward pass (Full stacktrace below).I built the model exactly how its done in train.py and my training loop looks like
The
model.forward
always fails with the below stacktrace. I've tried the forward pass with and without labels, similar results. Afterprepare_inputs_labels_for_multimodal
call, the inputs look like the following:Below is the full stacktrace and the model layers. What am I missing?
Model
Full StackTrace
The text was updated successfully, but these errors were encountered: