Support for older Python versions

[techthiyanes]

Could i know when PIP installed version to be rolled out? Also we are unable to locate (from tiktoken import _tiktoken) as well.

[mannasoumya]

Facing similar issue:
Using Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux:

ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'

[hauntsaninja]

@techthiyanes pip install tiktoken should just work. The project is available on PyPI, see https://pypi.org/project/tiktoken/

@mannasoumya Could you provide more details, e.g. maybe a script that reproduces the issue in a fresh environment? Are you installing from a wheel or from source? If from source, what version of setuptools, pip, setuptools-rust and do you have the build output?

[techthiyanes]

@techthiyanes pip install tiktoken should just work. The project is available on PyPI, see https://pypi.org/project/tiktoken/

@mannasoumya Could you provide more details, e.g. maybe a script that reproduces the issue in a fresh environment? Are you installing from a wheel or from source? If from source, what version of setuptools, pip, setuptools-rust and do you have the build output?

Thanks for mentioning about PIP details. While trying to install via colab using pip i have been getting below error.
ERROR: Could not find a version that satisfies the requirement tiktoken==0.1.1 (from versions: none)
ERROR: No matching distribution found for tiktoken==0.1.1

[hauntsaninja]

@techthiyanes One thing that could cause that message is if you're using an older Python; this initial release requires Python 3.9 or newer (last time I checked I think colab defaulted to an older Python version)

[techthiyanes]

Sure... Thank you for your response..i will try installing locally with 3.9.

[mannasoumya]

@techthiyanes pip install tiktoken should just work. The project is available on PyPI, see https://pypi.org/project/tiktoken/

@mannasoumya Could you provide more details, e.g. maybe a script that reproduces the issue in a fresh environment? Are you installing from a wheel or from source? If from source, what version of setuptools, pip, setuptools-rust and do you have the build output?

I have created a virtual environment of the above Python 3.10 and installed tiktoken via pip.

[abdelhadi-azouni]

got it to work using

python3.9 -m pip install tiktoken

then running my test.py with

python3.9 test.py

[hauntsaninja]

Just to clarify the state of this issue:

First, tiktoken currently only supports Python 3.9 and newer. This is the issue that techthiyanes and abdelhadi-azouni ran into. Future versions of tiktoken may add support for older Python. Thumbs up this comment if support for older Python is something you'd like.

[hauntsaninja]

The other thing discussed on this issue is that @mannasoumya is running into some undiagnosed issue that I cannot reproduce. The following works for me. What does the equivalent output for each of the following commands look like for you?

~/tmp λ python3.10 -m venv env            
~/tmp λ source env/bin/activate
(env) ~/tmp λ python3.10 -m pip install tiktoken
Collecting tiktoken
  Downloading tiktoken-0.1.1-cp310-cp310-macosx_10_9_x86_64.whl (725 kB)
...
(env) ~/tmp λ python3.10 -c 'import tiktoken; enc = tiktoken.get_encoding("gpt2"); print(enc.encode("hello world"))'                
[31373, 995]

[veered]

Support for older versions of Python would certainly be appreciated. Compatibility with 3.6 would be ideal, but 3.8 would also work. A lot of people won't be running 3.9.

[mannasoumya]

The other thing discussed on this issue is that @mannasoumya is running into some undiagnosed issue that I cannot reproduce. The following works for me. What does the equivalent output for each of the following commands look like for you?

~/tmp λ python3.10 -m venv env            
~/tmp λ source env/bin/activate
(env) ~/tmp λ python3.10 -m pip install tiktoken
Collecting tiktoken
  Downloading tiktoken-0.1.1-cp310-cp310-macosx_10_9_x86_64.whl (725 kB)
...
(env) ~/tmp λ python3.10 -c 'import tiktoken; enc = tiktoken.get_encoding("gpt2"); print(enc.encode("hello world"))'                
[31373, 995]

This seems to be working now. I think I had some issues while previously building Python from source. I rebuilt Python 3.10 from source and then created venv and installed tiktoken. Its working fine now. Thanks for the help.

[avideci]

Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux

Type "help", "copyright", "credits" or "license" for more information.
>>> import tiktoken
>>> tiktoken
<module 'tiktoken' from 'venv/lib/python3.8/site-packages/tiktoken/__init__.py'>
>>> tiktoken.get_encoding('gpt2')
<Encoding 'gpt2'>

I managed to make it work with python 3.8, and practically >= 3.7.
It takes some typing hints modification, they chose to use dict[...] instead of Dict, List, Set and FrozenSet types.
I'll open a PR for this one soon.

[avideci]

See https://github.com/openai/tiktoken/pull/14/files

To install in older python versions until that PR is merged, you can fork and run:

python3 setup.py build bdist_wheel
python -m pip install dist/tiktoken-0.1.1-cp38-cp38-linux_x86_64.whl 

Hope I helped.

[delip]

FWIW, I am on Python 3.9.12 and I see this error:

In [1]: import tiktoken
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import tiktoken

ModuleNotFoundError: No module named 'tiktoken'

[hauntsaninja]

I've added Python 3.8 support and released this in tiktoken 0.1.2.

I'm not currently planning on adding Python 3.7 support.

• Note that Python 3.7 will be end of life in five months, so consider upgrading
• Several packages have dropped support for 3.7, e.g. recent versions of numpy
• It looks like colab has moved to Python 3.8, which should resolve @techthiyanes issue from above

@delip I'll need more information to help you. Can you run the equivalent of the commands I mentioned in this comment: #1 (comment) and open a new issue with the output?