LLaMA Implementation

[zphang]

What does this PR do?

Implementation of LLaMA models (https://arxiv.org/abs/2302.13971). Model weights can be requested here. Weight conversion script is included.

Weights conversion can be run via:

python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights \
    --model_size 7B \
    --output_dir /output/path

Models can then be loaded via:

tokenizer = transformers.LLaMATokenizer.from_pretrained("/output/path/tokenizer/")
model = transformers.LLaMAForCausalLM.from_pretrained("/output/path/llama-7b/")

Example:

batch = tokenizer(
    "The primary use of LLaMA is research on large language models, including",
    return_tensors="pt", 
    add_special_tokens=False
)
batch = {k: v.cuda() for k, v in batch.items()}
generated = model.generate(batch["input_ids"], max_length=100)
print(tokenizer.decode(generated[0]))

Fixes #21796

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @younesbelkada

[ewof]

does this work with int8?

[zphang]

does this work with int8?

No idea! I haven't messed with int8 too much myself. It ought to be compatible with whatever is already supported in the HF models.

[YellowRoseCx]

nice work! thanks for the upload and I hope it gets pulled

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[zphang]

It looks like the tests which are currently failing are unrelated to the LLaMA code, so this should be good to review/use.

If folks can try it out (particularly with the larger, sharded models) and see if there are any issues, that will be helpful!

[USBhost]

It looks like the tests which are currently failing are unrelated to the LLaMA code, so this should be good to review/use.

If folks can try it out (particularly with the larger, sharded models) and see if there are any issues, that will be helpful!

At lest the convert script seems to work fine. I was able to convert 7B to 30B. I do not have enough ram to convert 65B.

[deepdiffuser]

Great work. thanks for putting this together

[USBhost]

After replacing transformers from Kobold with this PR I am able to load the shards as expected. Just I cant generate anything because Kobold still needs some changes.

[zsc]

does this work with int8?

No idea! I haven't messed with int8 too much myself. It ought to be compatible with whatever is already supported in the HF models.

Int8 seems not working but float16 is fine, in my hasty put-together test at https://github.com/zsc/llama_infer . Please throw a comment in case you find something!

[zsc]

@zphang I'm not able to get something like tokenizer = AutoTokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/") to work. Is this intentional or just leaving AutoTokenizer for future work?

[zphang]

@zphang I'm not able to get something like tokenizer = AutoTokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/") to work. Is this intentional or just leaving AutoTokenizer for future work?

What issue are you having / what is the error?

[oobabooga]

I have tested the code and these are my findings:

1. The conversion script works.
2. Loading the model works.
3. Loading the tokenizer with transformers.LLaMATokenizer.from_pretrained works.
4. Loading the tokenizer with AutoTokenizer.from_pretrained does not work and generates this error:

OSError: /tmp/converted/tokenizer/ does not appear to have a file named config.json. Checkout
'https://huggingface.co//tmp/converted/tokenizer//None' for available files.

5. The generated text seems to be incoherent. If I try these default values for the generation parameters:

model.generate(input_ids, eos_token_id=2, do_sample=True, temperature=1, top_p=1, typical_p=1, repetition_penalty=1, top_k=50, min_length=0, no_repeat_ngram_size=0, num_beams=1, penalty_alpha=0, length_penalty=1, early_stopping=False, max_new_tokens=200).cuda()

with this prompt:

Common sense questions and answers

Question: What color is the sky?
Factual answer:

I get

Common sense questions and answers

Question: What color is the sky?
Factual answer: Tags: python, django, django-models

Question: Using Django with multiple databases

I am attempting to use django with multiple databases, and I have the following code:

\begin{code}
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': ':memory:',
    },
    'db_one': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'db_one',
    },
    'db_two': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'db_two',
    },
}

It seems to me that prompts are being completely ignored.

6. Loading in 8-bit mode with load_in_8bit=True works.

[zsc]

This is OK: tokenizer = transformers.LLaMATokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/")

If using tokenizer = AutoTokenizer.from_pretrained("/data/llama/hf/7b/tokenizer/" then it will complain no "config.json".

OSError: /data/llama/hf/7b/tokenizer/ does not appear to have a file named config.json. Checkout 
'https://huggingface.co//data/llama/hf/7b/tokenizer//None' for available files.

I then hacked by softlinking /data/llama/hf/7b/tokenizer/special_tokens_map.json to /data/llama/hf/7b/tokenizer/config.json and it works. So maybe just rename?

Anyway, can now happily play with LLaMA in Hugging Face world and thanks for the great work!

[zphang]

Thanks for the comments. Looks like the saved tokenizer doesn't work for AutoTokenizer but works if you directly instantiate from LLaMATokenizer. Maybe one of the HF folks can chime in on the best way to address that.

The generated text seems to be incoherent. If I try these default values for the generation parameters:

Can you check the input_ids you're using to generate? The tokenizer currently adds both BOS and EOS tokens by default, and an EOS might cause the model to ignore your prompt.

Perhaps I can set EOS to not be added by default so it operates closer to expected behavior.

[oobabooga]

For this prompt:

'Common sense questions and answers\n\nQuestion: What color is the sky?\nFactual answer:'

these are the input_ids:

tensor([[    1, 13103,  4060,  5155,   322,  6089,    13,    13, 16492, 29901,
          1724,  2927,   338,   278, 14744, 29973,    13, 29943, 19304,  1234,
         29901,     2]], device='cuda:0')

I do not know how to interpret these numbers, but if there is an EOS token in that tensor and that token is causing the text generation to derail, changing that default would be valuable.

[zphang]

1 is BOS and 2 is EOS. Can you try without the last input id?

I also added an example in my PR message.

[oobabooga]

I confirm that doing this

    input_ids = input_ids[:, :-1]

to remove the last input id before calling model.generate(...) causes the text generation to become coherent:

Common sense questions and answers

Question: What color is the sky?
Factual answer: The sky is blue. The sky is blue, and it is a fact that it is blue. The sky is indisputably blue.

[wdyxwzyh]

@ longzhang418

Hello, I have a question about training llama. Does the current version of training not support the zero strategy of deepspeed? I added the deepspeed configuration file, but it doesn't seem to take effect. Looking for your reply, thank you~

Hi, we success use accelerate with ds_3_zero3_offload policy to train llama-13B in 4 A100s with memory 80GB(batch size=16 length=256 works for us ), firstly transfer 13B weights into hf weights format, then our accelerate config is in below:

compute_environment: LOCAL_MACHINE
deepspeed_config:
gradient_accumulation_steps: 1
gradient_clipping: 1.0
offload_optimizer_device: 'cpu'
offload_param_device: none
zero3_init_flag: true
zero3_save_16bit_model: true
zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
dynamo_backend: 'NO'
fsdp_config: {}
machine_rank: 0
main_training_function: main
megatron_lm_config: {}
mixed_precision: 'fp16'
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
use_cpu: false

The fsdp should works too, as the impl in standford repo.
significantly, we test 13B with ds0/ds1/ds2 on one A100 all failed because of CUDA memory.

Finally we failed on test some model pipeline methods, eg. fairscale pp, it's the next thing we need to do.

[haorannlp]

What's the motivation for the three special tokens tokenizer.pad_token, tokenizer.bos_token, tokenizer.eos_token = '' when converting the llama tokenizer?

[MrZilinXiao]

What's the motivation for the three special tokens tokenizer.pad_token, tokenizer.bos_token, tokenizer.eos_token = '' when converting the llama tokenizer?

Yeah, I am also quite confused. And I noticed even with add_special_tokens=True, LlamaTokenizer will not append an EOS token explicitly.

[ArthurZucker]

Just started working on a fix for the tokenizer conversion, will link the PR asap: #22402

[ucas010]

what's the version of huggingface-transformers for LLaMa ?

[hscspring]

When I am converting the llama to hf format (using convert_llama_weights_to_hf.py), each part of layer is the same size as the whole file
is anyone who met the same issue before?

python convert_llama_weights_to_hf.py  --input_dir models/ --model_size 7B --output_dir llama/fp32/

the subfiles (it is processing):

-rw-rw-r--. 1 wyd anaconda  26G Apr 12 15:55 pytorch_model-1-of-33.bin
-rw-rw-r--. 1 wyd anaconda  26G Apr 12 15:59 pytorch_model-2-of-33.bin
-rw-rw-r--. 1 wyd anaconda 129M Apr 12 15:59 pytorch_model-3-of-33.bin

the raw file:

-rw-rw-r--. 1 wyd anaconda  26G Apr 12 15:50 consolidated.00.fp32.pth

it's going right when i converted the llama to fp16.

[constanzafierro]

I think there might be something off with this model and tokenizer https://discuss.huggingface.co/t/llamatokenizerfast-returns-token-type-ids-but-the-forward-pass-of-the-llamamodel-does-not-receive-token-type-ids/42431 could someone take a look?

[ArthurZucker]

Hey! Yes this is fixed by #24042

[CoaLee]

@zphang Hi,
Developing a streaming(token by token) generation application, I faced the need for this functionality. I can't find its implementation with some minutes of searching back and forth. Could you point it out?

[ArthurZucker]

The add_prefix_space will be added in #25224