getting error while running poetry show

[harshachopra507]

I have installed poetry 1.4.0 version. I am getting the following error while running poetry show:

Command ['python', '-I', '-W', 'ignore', '-'] errored with the following return code 2

Error output:
Unknown option: -I
usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.


Input:
import sys

if hasattr(sys, "real_prefix"):
    print(sys.real_prefix)
elif hasattr(sys, "base_prefix"):
    print(sys.base_prefix)
else:
    print(sys.prefix)

[dimbleby]

Please do the things the issue template asks you to do: report the python version, the OS version, rerun with -vvv and include the output, etc etc etc

[gchamon]

from #7871

• Poetry version: 1.4.2
• Python version: 3.10.11
• OS version and name: Ubuntu 18.04
• pyproject.toml: https://gist.github.com/gchamon/c046bbb5340fc0926620e187ac360f31

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Somehow, running poetry install or poetry show, (like in issue #7854), poetry's utils/env.py resolves to python bin, which in the context of the execution seems to be python 2.7, which in turn has no option -I.

Complete execution of poetry install -vvv:

Loading configuration file /home/ubuntu/.config/pypoetry/config.toml

  Stack trace:

  1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1530 in _run
      1528│         try:
      1529│             if input_:
    → 1530│                 output = subprocess.run(
      1531│                     cmd,
      1532│                     stdout=subprocess.PIPE,

  CalledProcessError

  Command '['python', '-I', '-W', 'ignore', '-']' returned non-zero exit status 2.

  at /usr/lib/python3.10/subprocess.py:526 in run
       522│             # We don't call process.wait() as .__exit__ does that for us.
       523│             raise
       524│         retcode = process.poll()
       525│         if check and retcode:
    →  526│             raise CalledProcessError(retcode, process.args,
       527│                                      output=stdout, stderr=stderr)
       528│     return CompletedProcess(process.args, retcode, stdout, stderr)
       529│ 
       530│ 

The following error occurred when trying to handle this error:


  Stack trace:

  14  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  13  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  12  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  11  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  10  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:454 in _run_command
       452│ 
       453│         try:
     → 454│             self._event_dispatcher.dispatch(command_event, COMMAND)
       455│ 
       456│             if command_event.command_should_run():

   9  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/events/event_dispatcher.py:26 in dispatch
        24│ 
        25│         if listeners:
     →  26│             self._do_dispatch(listeners, event_name, event)
        27│ 
        28│         return event

   8  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/events/event_dispatcher.py:89 in _do_dispatch
        87│                 break
        88│ 
     →  89│             listener(event, event_name, self)
        90│ 
        91│     def _sort_listeners(self, event_name: str) -> None:

   7  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/console/application.py:299 in configure_env
       297│ 
       298│         env_manager = EnvManager(poetry, io=io)
     → 299│         env = env_manager.create_venv()
       300│ 
       301│         if env.is_venv() and io.is_verbose():

   6  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:895 in create_venv
        893│ 
        894│         cwd = self._poetry.file.parent
     →  895│         env = self.get(reload=True)
        896│ 
        897│         if not env.is_sane():

   5  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:745 in get
        743│                 return self.get_system_env()
        744│ 
     →  745│             return VirtualEnv(venv)
        746│ 
        747│         if env_prefix is not None:

   4  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1721 in __init__
       1719│         # from inside the virtualenv.
       1720│         if base is None:
     → 1721│             output = self.run_python_script(GET_BASE_PREFIX)
       1722│             assert isinstance(output, str)
       1723│             self._base = Path(output.strip())

   3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1508 in run_python_script
       1506│ 
       1507│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:
     → 1508│         return self.run(
       1509│             self._executable,
       1510│             "-I",

   2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1500 in run
       1498│     def run(self, bin: str, *args: str, **kwargs: Any) -> str | int:
       1499│         cmd = self.get_command_from_bin(bin) + list(args)
     → 1500│         return self._run(cmd, **kwargs)
       1501│ 
       1502│     def run_pip(self, *args: str, **kwargs: Any) -> int | str:

   1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1781 in _run
       1779│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
       1780│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
     → 1781│         return super()._run(cmd, **kwargs)
       1782│ 
       1783│     def get_temp_environ(

  EnvCommandError

  Command ['python', '-I', '-W', 'ignore', '-'] errored with the following return code 2
  
  Error output:
  Unknown option: -I
  usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
  Try `python -h' for more information.
  
  
  Input:
  import sys
  
  if hasattr(sys, "real_prefix"):
      print(sys.real_prefix)
  elif hasattr(sys, "base_prefix"):
      print(sys.base_prefix)
  else:
      print(sys.prefix)
  

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1545 in _run
      1541│                 return subprocess.call(cmd, stderr=stderr, env=env, **kwargs)
      1542│             else:
      1543│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
      1544│         except CalledProcessError as e:
    → 1545│             raise EnvCommandError(e, input=input_)
      1546│ 
      1547│         return decode(output)
      1548│ 
      1549│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

Changing

poetry/src/poetry/utils/env.py

Line 1225 in 3804a13

self._executable = "python"

to my expected python env interpreter (/home/ubuntu/.local/share/pypoetry/venv/bin/python) forces the python interpreter to be correctly derived and installations resumes.

Edit: even though installation continues, the environment is borked. It still defaults to system's python binaries, bypassing any lib installed by poetry.

[gchamon]

I created a fresh user in the same system and poetry works. I think something screwed my user installation.

[gchamon]

So what ended up working for me is erasing every trace of poetry from my system. First through the usual channels:

python3 -m pip uninstall poetry
curl -sSL https://install.python-poetry.org | python3 - --uninstall

Then using mlocate trace what was left of poetry in the system:

sudo updatedb
locate poetry

What I ended up removing was:

rm -r /home/ubuntu/.local/lib/python3.10/site-packages/poetry*
rm -r /home/ubuntu/.config/pypoetry
rm -r /home/ubuntu/.cache/pypoetry
rm /home/ubuntu/.local/bin/poetry

Then I could reinstall poetry. This time I opted for installing poetry using pip and python 3.9, which is more stable in my system (the collections.abc renaming is pulling my leg time and again):

pip3.9 install --user poetry

[ripperdoc]

For what it's worth, I have the same issue, on a MacOS where default system python is 2.7 but I installed poetry using python3 which is version 3.10. The installation of poetry should ensure it sticks with the same python3?

[crobison-ut]

Experiencing the same problem on Rocky 8.7, having uninstalled Poetry 1.1.14 (which was installed under Python 3.6.8, and working well):
curl -sSL https://install.python-poetry.org | python3 - --uninstall
..after which I installed Poetry 1.5.0 under Python 3.10.10:
curl -sSL https://install.python-poetry.org | python3.10

Running poetry update or poetry install generates the exact error given here, ultimately 'Unknown option: -I'.

I can take a look at the shebang in which poetry and see it's pointing to ~/.local/share/pypoetry/venv/bin/python. I can then either
~/.local/share/pypoetry/venv/bin/python -I
or
source ~/.local/share/pypoetry/venv/bin/activate and then python -I

and it works as expected.

I was not getting this error before -- it's not clear to me whether Poetry is now forcing the use of an older incompatible Python in a way it wasn't doing before, or whether this part of Poetry was always calling an old Python, but not using -I.

[crobison-ut]

Just discovered a new clue. I created a test user from scratch, and as that user repeated the installation above.
curl -sSL https://install.python-poetry.org | python3.10

I then created a test directory and ran poetry init inside, creating a simple pyproject.toml. Followed that with poetry update and it worked -- it created the virtualenv and lock file.

However, I had forgotten that Poetry defaults to creating virtualenvs under ~/.cache, and I vastly prefer them in-project. So I clean up and try again:

poetry env remove 3.10
poetry config virtualenvs.in-project true
poetry update

And now it fails, with 'Unknown option: -I'.

[dimbleby]

I guess this is a duplicate of #7957, just revealing itself in a different way. In both cases the result is that poetry is calling python instead of calling a particular python in a particular virtual environment

• on some systems that just fails outright because there is no such binary (Poetry 1.5 fails to install dependencies #7957)
• on others it finds the wrong python, a python27, and that goes wrong (this issue)

so it is fixed by the same fix

Edit: or anyway that's probably true for the virtualenvs.in-project variation per previous comment, I guess it can't be the case though for pre-1.5.0 examples

[hartzell]

Back in #8012, where I described poetry failing on my mac because it invoked python 2, I was asked to move the discussion here.

In #8012, someone said:

fail more explicitly if failing to find a good python.

What am I supposed to do when it fails? Don't use Poetry? Migrate to Linux?

I work with one of the commenters in the other thread and it seems like the reason that he doesn't have a problem is that he's been burned enough times that he does this wherever he goes.

$ ls -al $(which python)
lrwxr-xr-x  1 dennis.ritchie  staff  7 Apr 14 09:04 /Users/dennis.ritchie/.local/bin/python -> python3

That's a fine trick and an ok workaround, but if that's the official Poetry answer then we should put it in the FAQ.

Alternatively, would a PR that ran a bit of code to choose a modern enough interpreter name (e.g. try python, if it's v2 then try python3, if that doesn't work then fail gracefully) be welcomed?

Or perhaps find some way to remember and use the same python that was used to install poetry in the first place (in my case, python3)?

The same co-worker said:

I once read a thread where Guido was saying "Python 3 binary should always be python3"

I can't find his quote, but I can find oodles of bikeshedding arguments about it. Perhaps since Poetry requires 3.6 or newer, a PR that changed the name of the executable to python3 would be safe/safer.

[dimbleby]

you'll want to investigate why the call to get_system_env() (per the stack here) does not find a suitable python in this code

[hartzell]

you'll want to investigate

Cool and will do. Thanks for the lead!

[hartzell]

Ok, this is now a non-problem, though I don't know why.

I took @dimbleby's suggestion, set a break point, walked into the code, and found the really weird result that the virtual environment didn't have a bin dir, but rather a usr with a local with a bin dir (see below).
Fortunately I cut/paste the bits included below, so I'm not crazy, but.... I cleaned up (rm -rf /Users/georgehartzell/Library/Caches/pypoetry/) and tried to recreate it and....

Now everything works (starting from an uninstall and install of poetry v 1.5.0) and I can not get it to fail. While working along this today I installed the official Python .pkg file version of Python 3.11, uninstalled it using pkgutil and rm, and reinstalled via homebrew. Perhaps that changed something? Also had been playing with older versions of poetry, but can't recreate it that way either.

There are enough screen copies in this and the #8012 thread that I don't think I was hallucinating, but I can no longer recreate this.

If it happens again or I figure out what I changed, I'll pick the thread back up.

@dimbleby -- thank you for the pointers!

For posterity's sake, here's what I saw the first time I walked into the code.

---

Well, weird.

(Pdb) p bin_dir
PosixPath('/Users/georgehartzell/Library/Caches/pypoetry/virtualenvs/foo-XlONOBww-py3.11/bin')
(Pdb) p python_executables
[]

but

$ ls /Users/georgehartzell/Library/Caches/pypoetry/virtualenvs/foo-XlONOBww-py3.11/
lib        pyvenv.cfg usr
$

and

$ ls /Users/georgehartzell/Library/Caches/pypoetry/virtualenvs/foo-XlONOBww-py3.11/usr/local/bin
activate         activate.fish    activate.ps1     pip              pip3             python           python3.11       wheel-3.11       wheel3.11
activate.csh     activate.nu      activate_this.py pip-3.11         pip3.11          python3          wheel            wheel3
$

[hartzell]

Just did a bit of poking around because I hate feeling like I'm bonkers, it seems that I'm not the first person to end up with virtual environments that have a usr/local/bin instead of a bin dir, including one in a Poetry issue. I'm going to conclude that something was fishy in my Python installation and that something in the course of installing the official Python .pkg distribution over my brew install, then cleaning up, then re-installing the brew version got everything facing in the proper direction.

If you anyone else ends up here, these might be useful:

• Poetry attemps to use bin/python instead of local/bin/python in virtual env on debian systems #5925
• Python binaries in (venv)/usr/local/bin instead of (venv)/bin pypa/virtualenv#2319
• venv/usr/local/bin instead of venv/bin ? nanoporetech/katuali#10
• RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing pypa/build#215
• https://askubuntu.com/questions/1406304/virtualenv-installs-envs-into-local-bin-instead-of-bin

[trompx]

I use poetry 1.4.2 inside a docker container with asdf to handle python versions. Poetry was installed with python 3.8.10 (installed with asdf and set as global version).

I have a project with .tool-versions python 2.7.18 and also get this error when running poetry install.
I have POETRY_VIRTUALENVS_PREFER_ACTIVE_PYTHON=true and POETRY_VIRTUALENVS_IN_PROJECT=true.
Just tried the poetry 1.5.0 version and still get the same error.

If I switch .tool-versions to python 3.8.10, I don't get this error but another one:

The current project's Python requirement (>=2.7,<2.8 || >=3.4) is not compatible with some of the required packages Python requirement:
  - pandas requires Python >=3.7.1, so it will not be satisfied for Python >=2.7,<2.8 || >=3.4,<3.7.1
Because XXX depends on pandas (1.3.5) which requires Python >=3.7.1, version solving failed.

Which I don't get as I am using python 3.8.10.
Also, output of poetry debug info:

Poetry
Version: 1.5.0
Python:  3.8.10

Virtualenv
Python:         3.8.10
Implementation: CPython
Path:           /home/www-data/dev/proja/.venv
Executable:     /home/www-data/dev/proja/.venv/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.8.10
Path:       /home/www-data/.asdf/installs/python/3.8.10
Executable: /home/www-data/.asdf/installs/python/3.8.10/bin/python3.8

Is there any specific volume or env var I need to pass in the docker container while running the poetry command?

[abrennan-ams]

Same idea hartzell suggested

Would it make sense to prefer python3* over python in this code?

My thinking is OS that use python2.7 as default python could remain unchanged. If python3* doesn't exist, python would be used. This could be made a config parameter that would default to use python.

On the other hand, I can see how this could be an issue that needs to be resolved in the install environment and not in the code.

[codeADai]

I have the same problem.
My env is ubuntu 18.04, python3.9, poetry 1.5.0.
when I send poetry shell or poetry update that will return

Command ['python', '-I', '-W', 'ignore', '-'] errored with the following return code 2

Error output:
Unknown option: -I
usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.


Input:
import sys

if hasattr(sys, "real_prefix"):
    print(sys.real_prefix)
elif hasattr(sys, "base_prefix"):
    print(sys.base_prefix)
else:
    print(sys.prefix)

then today I updated the poetry version to 1.5.1
the problem is fixed, but I can't find which issue is related to this.
Thank.

[crobison-ut]

I guess we're talking about a bunch of different things in this thread. But FWIW, following up on my own previous comments here -- I upgraded to 1.5.1 and tried it again.

poetry self update
poetry config virtualenvs.in-project true
poetry env use python3.10

And it created the in-project virtualenv in .venv, no errors. Many thanks for the fix!

[ralbertazzi]

@dimbleby has #7977 fixed this too?

[dimbleby]

there are (at least) two things going on here, per #7854 (comment)

one of them will be fixed in 1.5.1, the other presumably won't.

the most likely lead on the other is #7854 (comment), saying that sometimes virtual environments get the wrong layout. This is a known problem on debian - #6371 - but some reports in here say that they're hitting it on MacOS which as far as I know is a surprise.

However for the only such reporter who has dug into it: the problem went away after some rounds of reinstalling things.

I think it's likely that (as of 1.5.1) this project is doing nothing wrong and there will therefore be no fix forthcoming: but perhaps this issue has some value anyway as a space for folk to tell each other how they fixed it for themselves.

(arguably that purpose would be just as well served by a closed issue!)