Skip to content

Commit

Permalink
Add Tiktokenizer link in "How to count tokens" (#604)
Browse files Browse the repository at this point in the history
This adds a link to Tiktokenizer webapp as another tool, in
addition to the OpenAI Tokenizer.
  • Loading branch information
EliahKagan authored Aug 28, 2023
1 parent a4913d3 commit 1b3ef07
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion examples/How_to_count_tokens_with_tiktoken.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
"\n",
"## How strings are typically tokenized\n",
"\n",
"In English, tokens commonly range in length from one character to one word (e.g., `\"t\"` or `\" great\"`), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., `\" is\"` instead of `\"is \"` or `\" \"`+`\"is\"`). You can quickly check how a string is tokenized at the [OpenAI Tokenizer](https://beta.openai.com/tokenizer)."
"In English, tokens commonly range in length from one character to one word (e.g., `\"t\"` or `\" great\"`), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., `\" is\"` instead of `\"is \"` or `\" \"`+`\"is\"`). You can quickly check how a string is tokenized at the [OpenAI Tokenizer](https://beta.openai.com/tokenizer), or the third-party [Tiktokenizer](https://tiktokenizer.vercel.app/) webapp."
]
},
{
Expand Down

0 comments on commit 1b3ef07

Please sign in to comment.