Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zipapp virtual environment creation fails if zipapp path is symlinked #1949

Closed
andyneff opened this issue Sep 25, 2020 · 14 comments
Closed

zipapp virtual environment creation fails if zipapp path is symlinked #1949

andyneff opened this issue Sep 25, 2020 · 14 comments
Assignees

Comments

@andyneff
Copy link

andyneff commented Sep 25, 2020

Issue

On a fresh new system, trying to use the zipapp to setup a virtualenv when the pyz files is referenced by a symlink, it fails saying KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

But it works on my machine?

Yes, if you ever used the virtualenv (zipapp?) successfully, it creates files in ~/Library/Application Support/virtualenv (on macOS) or ~/.local/share/virtualenv/ (on Linux) and the existence of these files (unzip and wheel) makes the zipapp work. By deleting that directory, you should be able to reproduce the problem. This was first discovered on travis-ci, where the machine is new every time.

Environment

  • OS: macOS 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64.
  • Python 3.7.7. (from brew), also tried 3.7.9

Also reproduced on

  • OS: fedora:31
  • Python 3.7.9

Output of the virtual environment creation

bash-3.2$ (cd /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo; python3 /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz -vvv --with-traceback /var/folders/g0/944401t11gz1jl7vxt
nzqs_m0001yj/T/foo/pipenv-2020.6.2)
206 setup logging to NOTSET [DEBUG report:43]
219 find interpreter for spec PythonSpec(path=/usr/local/opt/python/bin/python3.7) [INFO builtin:44]
219 proposed PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
219 accepted PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
224 filesystem is not case-sensitive [DEBUG info:28]
263 create virtual environment via CPython3Posix(dest=/private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2, clear=False, global=False) [INFO session:52]
263 write /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/pyvenv.cfg [DEBUG pyenv_cfg:34]
263     home = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
263     implementation = CPython [DEBUG pyenv_cfg:38]
263     version_info = 3.7.7.final.0 [DEBUG pyenv_cfg:38]
263     virtualenv = 20.0.31 [DEBUG pyenv_cfg:38]
263     include-system-site-packages = false [DEBUG pyenv_cfg:38]
264     base-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-exec-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-executable = /usr/local/opt/python/bin/python3.7 [DEBUG pyenv_cfg:38]
264 remove file /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:38]
264 symlink /usr/local/opt/python/bin/python3.7 to /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:44]
265 Attempting to acquire lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:270]
265 Lock 4396469136 acquired on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:274]
265 extract /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py to /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py [DEBUG zipapp:19]
267 Attempting to release lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:315]
267 Lock 4396469136 released on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:318]
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 168, in <module>
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 164, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/__init__.py", line 28, in cli_run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 46, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 53, in _create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/creator.py", line 157, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 86, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 88, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 91, in install_patch
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 27, in env_patch_text
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 102, in env_patch_text
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 81, in extract
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/util/zipapp.py", line 22, in extract

  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py", line 1431, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

Note: Notice /virtuale is missing from the KeyError, the exact length /private which is the difference between the TMPDIR and the real path of the TMPDIR on macOS

On Fedora, the last line becomes:

KeyError: "There is no item named '/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

Steps to reproduce

  1. docker run -it --rm fedora:31
  2. mkdir -p /tmp/foo/bar
  3. ln -s /tmp/foo/bar /tmp/foo/bar2
  4. curl -L -o /tmp/foo/bar/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/3.7/virtualenv.pyz
  5. python3 /tmp/foo/bar2/virtualenv.pyz /tmp/foo/blah

Here are some one-liners to reproduce the error

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'
docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

Not all values of REAL/SYM fail. It appears if the length of the symlink path and realpath differ, it fails. If they are the exact same length, no error occurs

Fail

REAL SYM
/tmp/foo/bar /tmp/bar/bar2
/tmp/foo/bar /tmp/foo/bc
/tmp/foo/car /tmp/foo/bb
/tmp/bb /tmp/aaa

Does not fail

REAL SYM
/tmp/foo /tmp/bar
/tmp/foo/bar /tmp/foo/abc
/tmp/foo/car /tmp/foo/bbb
@andyneff andyneff added the bug label Sep 25, 2020
andyneff added a commit to VisionSystemsInc/docker_recipes that referenced this issue Sep 25, 2020
andyneff added a commit to VisionSystemsInc/docker_recipes that referenced this issue Sep 28, 2020
@gaborbernat
Copy link
Contributor

Wouldn't this be fixed if you're using the copy method, instead of the symlink one? The symlink method cannot work if you're deleting the app-data folder from where we symlink. 🤔 ?

@andyneff
Copy link
Author

andyneff commented Oct 8, 2020

@gaborbernat Are you saying that a user should have to add flags when calling the zipapp?

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" --copies /tmp/blah'
docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" --symlink-app-data /tmp/blah'

both fail the same

@gaborbernat
Copy link
Contributor

I wonder why the copy fails, it should not, but will look into it. If you have free time to do it, and you can do it that would be even better 👍

@gaborbernat
Copy link
Contributor

Seems no longer an issue as no one reported it in 3 years.

@petamas
Copy link

petamas commented Jul 19, 2023

@gaborbernat: please reopen, I have just ran into this issue with virtualenv 20.24.0 on MacOS Catalina. (Why I'm still using Catalina is a long story.)

It seems to me that it's important that the .pyz is referred to through the symlink, i.e. if I cd into the directory, and only pass the name of the .pyz, then it works, but if I use the full path (which contains a symlink), then it fails.

@gaborbernat gaborbernat reopened this Jul 19, 2023
@gaborbernat
Copy link
Contributor

Please fill in a PR to fix it, we do not plan to work on this ourselves.

@petamas
Copy link

petamas commented Jul 19, 2023

Thanks for the quick reply! I have no idea where in the code the problem is, but I'll try to take a look.

(For now, I worked around it by calling os.path.realpath() on the path to the .pyz before calling it in my scripts.)

@petamas
Copy link

petamas commented Jul 19, 2023

Upon investigation of the code, it seems like addibg os.path.realpath to this line would solve the issue:

full_path = os.path.abspath(str(full_path))

ROOT has this function called, while full_path does not, so ROOT is not necessarily a prefix of full_path, making sub_file contain something weird. This also explains @andyneff 's observation about path lengths.

I'll try fixing it and PRing the fix later this week or next week.

petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023
petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023
petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023
petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023
@jamesnyao
Copy link

jamesnyao commented Jan 17, 2024

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

@bruceauyeung
Copy link

bruceauyeung commented Feb 26, 2024

I'm seeing this issue too when building electron
Python 3.11.8+chromium.31

$ '/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/bin/python3' '-B' '-I' '/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz' '--no-download' '--always-copy' '/home/testuser/.cache/.vpython-root/store/python_venv-478cu52ikjb86ilmh6apfpko0k/contents'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/__main__.py", line 167, in <module>
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/__main__.py", line 163, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/session.py", line 42, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/session.py", line 49, in _create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/creator.py", line 153, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 93, in create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 83, in create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 86, in install_patch
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 25, in env_patch_text
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 97, in env_patch_text
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 78, in extract
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/util/zipapp.py", line 19, in extract
    import {module}
                   ^
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/zipfile.py", line 1484, in getinfo
    raise KeyError(
KeyError: "There is no item named 'lenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

@bruceauyeung
Copy link

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

@bruceauyeung
Copy link

bruceauyeung commented Feb 29, 2024

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

#2613 merged after virtualenv-20.17.1 being released. So backporting those modifications to 20.17.1 can fix the problem. steps:

  1. cd /home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/
  2. mv virtualenv.pyz virtualenv.pyz.bak
  3. unzip virtualenv.pyz.bak -d virtualenv
  4. vi virtualenv/virtualenv/util/zipapp.py

replace contents in zipapp.py

full_path = os.path.abspath(str(full_path))
    sub_file = full_path[len(ROOT) + 1 :]

to

  full_path = os.path.realpath(os.path.abspath(str(full_path)))
  prefix = ROOT + os.sep
  assert full_path.startswith(prefix), f"full_path={full_path} should start with prefix={prefix}"
  sub_file = full_path[len(prefix) :]
  1. python3 -m zipapp virtualenv

@Zack4396
Copy link

Zack4396 commented Mar 1, 2024

Tried #1949 (comment) solution, and re-run gclient sync command, the errors are gone.

Thanks

@josegal
Copy link

josegal commented Apr 9, 2024

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

#2613 merged after virtualenv-20.17.1 being released. So backporting those modifications to 20.17.1 can fix the problem. steps:

  1. cd /home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/
  2. mv virtualenv.pyz virtualenv.pyz.bak
  3. unzip virtualenv.pyz.bak -d virtualenv
  4. vi virtualenv/virtualenv/util/zipapp.py

replace contents in zipapp.py

full_path = os.path.abspath(str(full_path))
    sub_file = full_path[len(ROOT) + 1 :]

to

  full_path = os.path.realpath(os.path.abspath(str(full_path)))
  prefix = ROOT + os.sep
  assert full_path.startswith(prefix), f"full_path={full_path} should start with prefix={prefix}"
  sub_file = full_path[len(prefix) :]
  1. python3 -m zipapp virtualenv

I just got this error on a fresh download of the chromium dev kit on ubuntu -- this workaround worked for me too but I had to move the virtualenv.pyz.bak file out of the way first... Thank you!!!

hubot pushed a commit to luci/luci-go that referenced this issue Apr 16, 2024
Use realpath for virtualenv.pyz to mitigate pypa/virtualenv#1949

Change-Id: I4ddfb78bc435f9791ccae5e87cf5c0da404a110a
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5449147
Reviewed-by: Brian Ryner <[email protected]>
Commit-Queue: Chenlin Fan <[email protected]>
Auto-Submit: Chenlin Fan <[email protected]>
hubot pushed a commit to luci/luci-go that referenced this issue Apr 16, 2024
This reverts commit 988aaae.

Reason for revert: Change is breaking luci-go to infra autoroller. https://ci.chromium.org/ui/p/infra/builders/try/infra-try-jammy-64/b8750527694180069633/overview

Original change's description:
> vpython: use realpath when invoking virtualenv
>
> Use realpath for virtualenv.pyz to mitigate pypa/virtualenv#1949
>
> Change-Id: I4ddfb78bc435f9791ccae5e87cf5c0da404a110a
> Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5449147
> Reviewed-by: Brian Ryner <[email protected]>
> Commit-Queue: Chenlin Fan <[email protected]>
> Auto-Submit: Chenlin Fan <[email protected]>

Change-Id: I37d27f0fd80d93f722ed000985fa5b95f51c8c6d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5458911
Commit-Queue: Rubber Stamper <[email protected]>
Bot-Commit: Rubber Stamper <[email protected]>
Auto-Submit: Randy Maldonado <[email protected]>
hubot pushed a commit to luci/luci-go that referenced this issue Apr 17, 2024
This is a reland of https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5449147
with fix for python2.7.

Use realpath for virtualenv.pyz to mitigate pypa/virtualenv#1949

Change-Id: I4bf2f527d1a90577d25194e05bb2de00bc1d71ef
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5458472
Commit-Queue: Brian Ryner <[email protected]>
Reviewed-by: Brian Ryner <[email protected]>
Auto-Submit: Chenlin Fan <[email protected]>
HandSonic pushed a commit to HandSonic/virtualenv that referenced this issue May 11, 2024
HandSonic added a commit to HandSonic/virtualenv that referenced this issue May 11, 2024
HandSonic added a commit to HandSonic/virtualenv that referenced this issue May 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants
@gaborbernat @bruceauyeung @andyneff @petamas @jamesnyao @josegal @Zack4396 and others