-
Notifications
You must be signed in to change notification settings - Fork 758
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Token ID Out of Range & Indexing Assertion Errors During Training #1609
Comments
Firstly, please revoke your token and edit it out of the issue so that hackers won't use it and pay you for their huggingface API usage. Secondly, it seems like the error is that The issue was probably caused by an error in the tokenizer files. Please report the bug to the llama repo |
Thanks for telling me about the HF token, and appreciate the quick response about the issue! |
It seems like there are two main issues here:
|
Your vocab_size should just be # Load tokenizer
model_name = 'meta-llama/Meta-Llama-3.1-8B-Instruct'
tokenizer = AutoTokenizer.from_pretrained(model_name, token=hf_token, use_fast=True)
-# Get the correct vocabulary size
-vocab_size = len(tokenizer.get_vocab())
if tokenizer.pad_token is None:
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
+# Get the correct vocabulary size after adding padding
+vocab_size = len(tokenizer.get_vocab()) |
Title: Token ID Out of Range & Indexing Assertion Errors During Training
Description:
I'm encountering several issues while training a model using the Meta-Llama-3.1-8B-Instruct tokenizer and dataset processing script. The main issues are as follows:
Token ID Out of Range:
During tokenization, I'm consistently receiving the following warning:
This occurs even after attempting to handle out-of-range token IDs by capping them at the maximum valid token ID (
127999
). This issue might be affecting the overall model performance and data integrity.Indexing Assertion Error:
When generating the training split, the following error is triggered:
This assertion failure suggests that there might be an issue with how indices are being selected during the training process, potentially due to misaligned tensor dimensions or out-of-range indices.
Code:
Here is the script I'm using for tokenization and dataset processing:
Minimal Reproducible Example:
Here is a minimal code example to reproduce the token ID out-of-range issue:
Output:
Steps to Reproduce:
Indexing.cu
assertion error during the generation of the training split.Environment:
Expected Behavior:
Token IDs should be within the valid range after tokenization. The training process should proceed without assertion errors, and there should be no conflicts between gradient checkpointing and caching.
Additional Context:
The data being processed includes a mix of unicode and non-unicode characters. The script attempts to clean the data by removing special characters and non-unicode sequences. Despite these precautions, the issues mentioned above persist.
Any guidance on resolving these issues or insights into potential causes would be greatly appreciated.
The text was updated successfully, but these errors were encountered: