Load pre-trained tokenizer from memory

[HaoboGu]

Hello guys,

tokenizers::Tokenizer has two methods to create a pre-trained tokenizer: Tokenizer::from_pretrained and Tokenizer::from_file. It's quite common that we need to download tokenizer.json from remote, now we have to save the remote data to a local file and then load the tokenizer from local file. We also have to delete it after loading.

If we have something like Tokenizer::from_memory(data_bytes: &[u8]), we can just download the remote tokenizer to memory and then load it directly. It's quick and safe, you'll never need to delete the local file after using.

[Narsil]

This is a totally reasonable thing to ask.

Actually if you are in pure Rust, it already supports serde so

let tokenizer: Tokenizer = serde_json::from_slice(&data_bytes).unwrap(); should work out of the box.

That being said adding a new function is definitely something we could do. from_bytes or from_slice seem like better names considering std but overall would still be a nice addition IMHO.