WARNING: Unused parameter in LoRA state dict

[iclgg]

Converts an RWKV model checkpoint in PyTorch format to an rwkv.cpp compatible file using convert_pytorch_to_ggml.py.
Get a LoRA checkpoint with https://github.com/Blealtan/RWKV-LM-LoRA.
Merges a LoRA checkpoint in PyTorch format (.pth) into an rwkv.cpp model file using merge_lora_into_ggml.py.
Warnings like that "Unused parameter in LoRA state dict blocks.13.att.receptance.lora_B(att.key.lora_A、att.value.lora_A、att.receptance.lora_A、ffn.key.lora_A、ffn.receptance.lora_A、ffn.value.lora_A、att.key.lora_B、att.value.lora_B、att.receptance.lora_B、ffn.key.lora_B、ffn.receptance.lora_B、ffn.value.lora_B)" were printed during the merge .
Using the merged model is poor and does not reflect the effect of lora.
Why does this happen?

[saharNooby]

If possible, can you provide the used LoRA checkpoint?

[iclgg]

If possible, can you provide the used LoRA checkpoint?

Now I cannot access the computer, so I cannot provide the LoRA file for the time being. Do you know what might be causing this?

[saharNooby]

Probably the mistake in the merge script. It expects one format, but gets the other. I need an example LoRA checkpoint (does not even need to be trained) to be able to fix the script.

[iclgg]

https://huggingface.co/iclgg/rwkv-lora/tree/main
This is a LoRA checkpoint based on RWKV-4-Raven-3B-v9x-Eng49%-Chn50%-Other1%-20230417-ctx4096.pth.
lora_alpha=32 lora_r=8 precision=fp16 --lora_parts=att,ffn,time,ln

[saharNooby]

@iclgg Thanks for the file! Can you please test this version of the merge script?

[iclgg]

@iclgg Thanks for the file! Can you please test this version of the merge script?

It works well! Thank you.