-
Notifications
You must be signed in to change notification settings - Fork 753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FAQ (please read before opening a new issue) #98
Comments
This was referenced Apr 5, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Usage help
Check out this awesome tokeniser app https://tiktokenizer.vercel.app/ built by Diagram!
Check out the OpenAI cookbook!
In particular, the following are great examples of using
tiktoken
:Other useful resources:
Support for other languages
Publishing an official NPM package
I have no current plans to do this, but it is likely that in the future OpenAI will provide an official package. Discuss at #22
Publishing an official Rust crate
In the future when I have time, I will publish the core BPE logic as its own crate. Discuss at #24
I'm short on time currently, but in the longer term I want to explore ways to make maintaining downstream bindings easier. Apologies to the people who've reached out about this — y'all are great, sorry for being a bottleneck.
Packaging
Issues with pyinstaller
I've never used pyinstaller, but people report
--hidden-import=tiktoken_ext.openai_public --hidden-import=tiktoken_ext
helps make things work. See #43 for more details.Wheels for other platforms
I'm reluctant to publish wheels for additional platforms, particularly if I can't easily build or test code for them. I'm unlikely to do this unless there's significant interest. Please add reactions to the following issues if you'd like to see those wheels. No promises, but I'll reopen any issue that gets >10 reactions.
The text was updated successfully, but these errors were encountered: