Remove duplicate words in docstring (openai#3)

xcharita · Dec 16, 2022 · 4226a6c · 4226a6c
1 parent ab3688a
commit 4226a6c
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/tiktoken/core.py b/tiktoken/core.py
@@ -59,7 +59,7 @@ def encode(
  ) -> list[int]:
  """Encodes a string into tokens.
 
- Special tokens are tokens are artificial tokens used to unlock capabilities from a model,
+ Special tokens are artificial tokens used to unlock capabilities from a model,
  such as fill-in-the-middle. So we want to be careful about accidentally encoding special
  tokens, since they can be used to trick a model into doing something we don't want it to do.