Embed files

[katsu560]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

sync ggml's PR: Embed yolo files #831 (ggerganov/ggml#831)

I added new script gguf-addfile.py to ggml. so I need sync llama.cpp side.
I closed gguf : embed files to gguf model file #7392.

[ggerganov]

After the CI is fixed and the file renamed we can merge

[ggerganov]

Use underscore in the python filenames: gguf-addfile.py -> gguf_add_file.py

[katsu560]

Thank you for checking.
I renamed the script to gguf_add_file.py.

[mofosyne]

If possible, it would be helpful if there was a readme or an entry in the wiki or comments in the code explaining how the file is being embedded and it's structure in the kv store... e.g. is there a structure for storing the file name? Is it stored as a dummy tensor?

[compilade]

@mofosyne The embedded files seem to be stored as tensors, so this means they must be loaded on model load, and so the embedded files cannot be optional metadata as in #8602.

(using tensors was suggested in ggerganov/ggml#831 (review) as a concern for the size of the metadata)

This limits the use-cases for embedded files to required files (no extra unhandled files), which might or might not be intended.

I don't really have a strong opinion on this, either way is fine, as long as the tradeoffs are known.

[compilade]

Quantized tensors won't have the correct shape otherwise. See #7483.

Suggested change

	# Dimensions are written in reverse order, so flip them first
	shape = np.flipud(tensor.shape)
	writer.add_tensor_info(tensor.name, shape, tensor.data.dtype, tensor.data.nbytes, tensor.tensor_type)
	writer.add_tensor_info(tensor.name, tensor.data.shape, tensor.data.dtype, tensor.data.nbytes, tensor.tensor_type)

[compilade]

It's ignored anyway because raw_dtype is specified, but I think the Numpy type given should at least be similar.

Suggested change

	writer.add_tensor_info(path, dims, np.float16, data_len, raw_dtype)
	writer.add_tensor_info(path, dims, np.int8, data_len, raw_dtype)