[Request] Support non standard pip executable locations in the plugin manager

[j0hax]

What were you doing?

1. Enable OctoPrint in NixOS via services.octoprint.enable
2. Add pip to the OctoPrint systemd unit's PATH with systemd.services.octoprint.path = [ pkgs.python3Packages.pip ];
3. Error that pip was not found

What did you expect to happen?

Pip should be found via the PATH, that is, under my current NixOS system, /nix/store/lg8bzxkn9af1bky6dxvbyw2gm9rzr9rr-python3.8-pip-20.1.1/bin

What happened instead?

Pip is searched for in the Python install directory, which does not include Pip on NixOS: /nix/store/ysfhjsvlrc7ra5qnsghmnwkk9p45jb1n-python3-3.8.5-env/bin.

I highly suspect this to be an issue in the function autodetect_pip() in /src/octoprint/util/pip.py, which relies on sys.executable to find pip.

Did the same happen when running OctoPrint in safe mode?

Yes.

Version of OctoPrint

1.4.0

Operating System running OctoPrint

NixOS 20.09 (unstable branch)

Printer model & used firmware incl. version

N/A

Browser and version of browser, operating system running browser

N/A

Link to octoprint.log

https://gist.github.com/j0hax/f5c85c1e7bd164e60289bd4e824b7132

Link to contents of terminal tab or serial.log

N/A

Link to contents of Javascript console in the browser

N/A

Screenshot(s)/video(s) showing the problem:

N/A

I have read the FAQ.

[foosel]

I highly suspect this to be an issue in the function autodetect_pip() in /src/octoprint/util/pip.py, which relies on sys.executable to find pip.

It is indeed. I'm going to reclassify this as a feature request to support non-standard pip locations for the plugin manager to use.

[foosel]

Ready for 1.5.0.

A globally configured local pip command can be set via server.commands.localPipCommand in the config. I've not exposed that on the UI since it could really mess up people's setups and should only ever be needed by people with extremely rare setups like yours who hopefully know what they are doing.

[foosel]

1.5.0 has just been released.

[MichaIng]

@foosel
It does not seem to be respected by all invocations. The plugin manager does, but the OctoPrint self-update still uses the default pip command. Also during startup, while first the localPipCommand is used and testballoon install used to detect if the target dir can be written to by pip or not, at a later stage LocalPipCaller's _check_pip_setup function is called which uses the default pip and does an own writable check (not via testballoon install). This overrides the custom pip command as well as the previously done writable check, leading to a wrong pip invocation.

def create_pip_caller(command=None, **kwargs):
    if command is None:
        return LocalPipCaller(**kwargs)
    else:
        return PipCaller(configured=command, **kwargs)

It seems to be that create_pip_caller is first called with a correctly passed configured custom pip command and again once without passing the custom pip command:

Dec 13 16:55:01 VM-Buster octoprint[6186]: 2020-12-13 16:55:01,067 - octoprint.util.pip - DEBUG - Going to figure out pip's version
Dec 13 16:55:01 VM-Buster sudo[6200]: octoprint : TTY=unknown ; PWD=/ ; USER=root ; ENV=TESTBALLOON_OUTPUT= ; COMMAND=/usr/local/bin/pip3 --version
Dec 13 16:55:01 VM-Buster sudo[6200]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 13 16:55:01 VM-Buster sudo[6200]: pam_unix(sudo:session): session closed for user root
Dec 13 16:55:01 VM-Buster octoprint[6186]: 2020-12-13 16:55:01,537 - octoprint.util.pip - INFO - Version of pip is 20.3.1
Dec 13 16:55:01 VM-Buster sudo[6205]: octoprint : TTY=unknown ; PWD=/usr/local/lib/python3.7/dist-packages/octoprint/util/piptestballoon ; USER=root ; ENV=TESTBALLOON_OUTPUT=/tmp/tmpa2f0qg2g ; COMMAND=/usr/local/bin/pip3 install .
Dec 13 16:55:01 VM-Buster sudo[6205]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 13 16:55:03 VM-Buster sudo[6205]: pam_unix(sudo:session): session closed for user root
Dec 13 16:55:03 VM-Buster octoprint[6186]: 2020-12-13 16:55:03,012 - octoprint.util.pip - INFO - pip installs to /usr/local/lib/python3.7/dist-packages/ (writable -> yes), --user flag needed -> no, virtual env -> no
Dec 13 16:55:03 VM-Buster octoprint[6186]: 2020-12-13 16:55:03,012 - octoprint.util.pip - INFO - ==> pip ok -> yes
...
Dec 13 16:55:05 VM-Buster octoprint[6186]: 2020-12-13 16:55:05,291 - octoprint.util.pip - DEBUG - Going to figure out pip's version
Dec 13 16:55:05 VM-Buster octoprint[6186]: 2020-12-13 16:55:05,291 - octoprint.util.pip - DEBUG - Using cached pip version information for /mnt/dietpi_userdata/octoprint/pip3
Dec 13 16:55:05 VM-Buster octoprint[6186]: 2020-12-13 16:55:05,292 - octoprint.util.pip - DEBUG - Using cached pip setup information for /mnt/dietpi_userdata/octoprint/pip3
Dec 13 16:55:05 VM-Buster octoprint[6186]: 2020-12-13 16:55:05,297 - octoprint.plugins.softwareupdate - INFO - Minimum free storage across all update relevant locations is 959.4MB. That is considered sufficient for updating.
Dec 13 16:55:05 VM-Buster octoprint[6186]: 2020-12-13 16:55:05,297 - octoprint.plugins.softwareupdate - INFO - Fetching check overlays from https://plugins.octoprint.org/update_check_overlay.json
Dec 13 16:55:05 VM-Buster octoprint[6186]: 2020-12-13 16:55:05,346 - octoprint.plugins.pluginmanager - INFO - Loaded plugin repository data from disk, was still valid
Dec 13 16:55:06 VM-Buster octoprint[6186]: 2020-12-13 16:55:06,303 - octoprint.plugins.pluginmanager - INFO - Loaded notice data from disk, was still valid
Dec 13 16:55:06 VM-Buster octoprint[6186]: 2020-12-13 16:55:06,492 - octoprint.util.pip - INFO - Using "/usr/bin/python3 -m pip" as command to invoke pip
Dec 13 16:55:06 VM-Buster octoprint[6186]: 2020-12-13 16:55:06,493 - octoprint.util.pip - DEBUG - Going to figure out pip's version
Dec 13 16:55:06 VM-Buster octoprint[6186]: 2020-12-13 16:55:06,941 - octoprint.util.pip - INFO - Version of pip is 20.3.1
Dec 13 16:55:06 VM-Buster octoprint[6186]: 2020-12-13 16:55:06,942 - octoprint.util.pip - INFO - pip installs to /usr/lib/python3/dist-packages (writable -> no), --user flag needed -> yes, virtual env -> no
Dec 13 16:55:06 VM-Buster octoprint[6186]: 2020-12-13 16:55:06,942 - octoprint.util.pip - INFO - ==> pip ok -> yes

As can be seen, the whole effort is to make OctoPrint calling pip via sudo to allow managing a global/system-side instance via web UI plugins management and updater. I also found pip_sudo variables in the code but setting localPipCommand: sudo /usr/local/bin/pip3 or such makes it being ignored (again only default pip used). /bin/sudo again is not detected by if pip_command is not None and pip_command.startswith("sudo "):.

[MichaIng]

Just verified by the log output on the web UI.
The plugins manager uses the configured custom pip command:

/mnt/dietpi_userdata/octoprint/pip3 --disable-pip-version-check install file:https:///tmp/tmprhx95vxb/OctoPrint-Autoscroll-master.zip --no-cache-dir

But the OctoPrint self-update ignores it and uses the default:

/usr/bin/python3 -m pip --disable-pip-version-check install https://github.com/OctoPrint/OctoPrint/archive/1.5.2.zip --no-cache-dir --user

[cp2004]

@MichaIng can you open a bug report, since this issue has been closed & the original request was solved, even if it seems broken, life is easier if we can separate it.

[MichaIng]

New issue opened: #3892