add jtokkit to How_to_count_tokens_with_tiktoken.ipynb

nickduran · Apr 28, 2023 · 4d59baf · 4d59baf
1 parent 297c534
commit 4d59baf
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/examples/How_to_count_tokens_with_tiktoken.ipynb b/examples/How_to_count_tokens_with_tiktoken.ipynb
@@ -37,6 +37,7 @@
  "For `cl100k_base` and `p50k_base` encodings:\n",
  "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n",
  "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken)\n",
+ "- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n",
  "\n",
  "For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",
  "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n",