Unresolved import problems

[floli]

Hello,

my Python LSP configuration is:

(use-package lsp-python-ms
  :ensure t
  :config (progn
            (setq lsp-python-ms-extra-paths '("/path_to_package/mypackage/src/")))
  :hook (python-mode . lsp))

Now I have a line in a python file that gives an unresolved import:

import mypackage.helper.glog as log

I validate that path above should fix the unresolved import by

$ unset PYTHONPATH 
$ python
Python 2.7.17 (default, Apr 15 2020, 17:20:14) 

>>> import mypackage.helper.glog as log
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named mypackage.helper.glog

>>> import sys
>>> sys.path.append("/path_to_package/mypackage/src/")

>>> import mypackage.helper.glog as log
>>> [no error message]

Still, Python LSP gives me an unresolved import 'mypackage'

Versions:

• GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.32) of 2020-05-07
• LSP as of today from MELPA

What can still be wrong at my setup?

Thanks!

[seagle0128]

lsp-python-ms-extra-paths is buffer-local. Can you check its value in buffer?

[floli]

Yes, it is set.

lsp-python-ms-extra-paths is a variable defined in ‘lsp-python-ms.el’.
Its value is
("/path_to_package/mypackage/src/")

[seagle0128]

Can you provide example files for troubleshootings?

[floli]

Sorry, just tried to reproduce on a vanilla example, but was unable to. It always worked as expected. However, at my original project it still fails.

I tried to delete my lsp-sessions and the ~/.emacs.d/.cache directory. LSP now asks to reinstall the server and fails with the same message I have described in the other issue.

Sorry, no helpful news. I keep trying and try to investigate.

[seagle0128]

Updating to the latest version will address the downloading issue.

[floli]

I think I am able to reproduce this issue. I have attached a simple python project:
mspyls.zip

mspyls <- git root there, so LSP asks to add this project root
    init.el <- I tested it with emacs -q --load init.el
    subdir/ <- that's important
    lib/hello.py
    src/main.py <- imported from lib import hello

it works (runs) when I set the PYTHONPATH and run python, but not when setting lsp-python-ms-extra-paths to the same value (unresolved import and no jumping to definition).

[seagle0128]

Please update to the latest and use this setting:

(setq-default lsp-python-ms-extra-paths '("/Users/vincent/Desktop/mspyls/subdir"))

[floli]

Hello,
unfortunately using lsp-python-ms-20200520.1609/ from elpa, I am still not successful. Updated config is:

(setq-default lsp-python-ms-extra-paths '("/lhome/lindnfl/scratch/mspyls/subdir"))
(use-package lsp-python-ms
  :ensure t
  :config (progn
            (setq-default lsp-python-ms-extra-paths
                  '("/lhome/lindnfl/scratch/mspyls/subdir")))
  :hook (python-mode . lsp))

I use setq-default twice to make sure it's set at the earliest occasion. Using my small example above, I still get the unresolved import.

[failable]

Same issue here. Although I use setq since lsp-python-ms-extra-paths is buffer local but using setq-default still no help. The variable is indeed passed into lsp-python-ms--extra-init-params but it is indeed not in :searchPaths of the returned value.

Setting PYTHONPATH does work, but is not that convenient.

[seagle0128]

@floli @Isolet Please check the value of (lsp--suggest-project-root). And set lsp-auto-guess-root to nil and retry. Or set the correct project root.

Remember to restart lsp server.

[failable]

Nothing changes. With lsp-auto-guess-root is nil or t, (lsp--suggest-project-root) return the right project root thought.

As I mentioned before,

The variable is indeed passed into lsp-python-ms--extra-init-params but it is indeed not in :searchPaths of the returned value.

[seagle0128]

It does work on my both laptops, with the sample project mspyls.zip.

[failable]

Yes, that works since lib has been packed into a module. However, if one puts a def hello_world in utils.py within the same directory of main.py, he just can neither import it in main.py nor jump to its definition. A project can contain modules, and example/script files which are not necessary to be a module.

[seagle0128]

I do believe you didn't set the proper workspace root. The behavior is same as VSCode exactly. Read the manual of lsp-mode. Actually if you set the workspace root properly, it's not necessary to set lsp-python-ms-extra-paths in this case.
You'd better add .projectile file to mspyls/subdir/ to make the things right. Also set the workspace root to mspyls/subdir/ interactively (press i typically in prompts of lsp-mode).

Here is my screenshot:

[failable]

I do believe you didn't read the comments carefully. The project is mspyls not mspyls/any-other-thing as mentioned by @floli , i.e, the subdir is important. The issue is shadowed if workspace root is set to the sub-directory. There maybe several sub-packages/modules, e.g. lib1, lib2... and script directories e.g. src, src/demo1, src/demo2, src1, src2... in mspyls.

mspyls
├── lib
│   ├── __init__.py
│   └── hello.py
├── lib2
│   ├── __init__.py
│   └── world.py
└── src
    ├── demo1
    │   ├── main1.py
    │   └── utils1.py
    ├── main.py
    └── utils.py

Why would I have to set all sub-directories to individual workspaces?

Any script in src* and all their sub-directories can resolve imports from modules lib*, that's correct behavior of lsp and vscode as stated in the document

The language server treats the workspace root (i.e. folder you have opened) as the main root of user module imports.

However, scripts in sub-directories of src* can not resolve imports from scripts in the same directory(e.g. import functions fromsrc/demo1/utils1.py in src/demo1/main1.py. This can be solved by setting

"python.autoComplete.extraPaths": [
      "src/demo1"
]

in vscode. However adding the same path to lsp-python-ms-extra-paths does not solve the issue.

This is exactly the same example given by @floli . One just can not import hello from subdir/lib in subdir/src/main if workspace root is mspyls .

[seagle0128]

@Isolet Did you try with the example provided by @floli ? If you set root to mspyls, it definitely fails to auto-complete, whatever lsp-python-ms or vscode. In that case, the root should be mspyls/subdir.

mspyls
├── lib
│   ├── __init__.py
│   └── hello.py
├── lib2
│   ├── __init__.py
│   └── world.py
└── src
    ├── demo1
    │   ├── main1.py
    │   └── utils1.py
    ├── main.py
    └── utils.py

In the case above, I agree the root should mspyls, and the auto-complete should work well. Otherwise please check your configurations of lsp-mode. Do you set project root correctly while seeing this? Try different configurations.

"mspyls is not part of any project. Select action:
i ==>Import project root %s.
I ==>Import project by selecting root directory interactively.
d ==>Do not ask again for the current project by adding %s to lsp-session-folders-blacklist.
D ==>Do not ask again for the current project by selecting ignore path interactively.
n ==>Do nothing: ask again when opening other files from the current project."

Generally there is no difference between lsp-mode and vscode since they are using gopls with similar settings.

[wsw0108]

modify lsp-python-ms.el like below fix this problem for me(add python.autoComplete.extraPaths to lsp custom settings).

(lsp-register-custom-settings
 `(("python.analysis.cachingLevel" lsp-python-ms-cache)
   ("python.analysis.errors" lsp-python-ms-errors)
   ("python.analysis.warnings" lsp-python-ms-warnings)
   ("python.analysis.information" lsp-python-ms-information)
   ("python.analysis.disabled" lsp-python-ms-disabled)
   ("python.autoComplete.extraPaths" lsp-python-ms-extra-paths)
   ("python.analysis.autoSearchPaths" ,(<= (length lsp-python-ms-extra-paths) 0) t)))

I set lsp-python-ms-extra-paths in .dir-locals.el. And python.analysis.autoSearchPaths send to LSP Server will be true(found in lsp log by setting setq lsp-log-io t), cause lsp-python-ms-extra-paths not used by LSP Server.

https://github.com/Microsoft/python-language-server/blob/master/src/LanguageServer/Impl/LanguageServer.Configuration.cs#L204-L215

https://github.com/Microsoft/python-language-server/blob/master/src/LanguageServer/Impl/SearchPaths/AutoSearchPathFinder.cs#L29-L32

setq-default should work as expected, but I think it's not convient.

[seagle0128]

@wsw0108 Thanks! I will merge the settings. lsp-python-ms-extra-paths is a buffer-local var, and should work fine in .dir-locals.el` now.

@Isolet @floli please try the latest version. Remember to set the project root properly as well.

[wsw0108]

@seagle0128
Sorry, the value passed to python.autoComplete.extraPaths should be vector, not list.
If it is list, latest lsp-mode will not be able to serialize it.

[seagle0128]

@wsw0108 Thank you. I've committed a patch to make lsp-python-ms-extra-paths to a vector. Please test.

[wsw0108]

@seagle0128 I have create a PR for this, please review it.

[wztdream]

I suffer from this issue too, but in my case the root cause is the root folder setting is not correct. The method to test what is the root folder for a file is M: (lsp-workspace-root), this function is not interactive, I think this is quite misleading. It seems there are several definition of root folder but they are not consistent. lsp-mode manual is not clear too.

You need to trun on the synchronize treemacs projects with lsp-mode workspace folders follow lsp-treemacs manul

[tm-schwartz]

I had a weird situation where only some of my installed libraries were resolved, as in numpy was found but h5py and sklearn were not. I already had a .env file setup for use with nvim/coc so I tried setting setq lsp-python-ms-parse-dot-env-enabled to t but still was having issues. Checked logs for lsp and found my PYTHONPATH was being read from the .env file but not correctly. For example, searchPaths in logs was showing "/my/workspace/root/\"/my/.env/python/path/"". To fix this I had to add a colon to the beginning of my PYTHONPATH string in my .env file like PYTHONPATH=":/path/...". I guess either the colon is left off when concatenating the buffer-string or its expected for there to be a leading colon.

[TiFaBl]

i run into this issue because i manage my python virtualenvironments through pyenv. Since this took me a while to get correct, i will try to summarize my findings.
To have lsp-python-ms recognize the imports and provide auto completion, it was not enought to pyvenv-workon or pyvenv-activate before loading or restarting the lsp workspace but i have to either use the .env or .dir-locals.el approach.

1. Using .env
   configure lsp-ython-ms to parse the .env file by setting lsp-python-ms-parse-dot-env-enabled t. An example with use-package:

(use-package lsp-python-ms
  :straight t
  :init (setq lsp-python-ms-auto-install-server t
	      lsp-python-ms-parse-dot-env-enabled t)
  :hook (python-mode . (lambda ()
                          (require 'lsp-python-ms)
                          (lsp))))  ; or lsp-deferred

In the .env file in the project root, add
PYTHONPATH=/home/user/.pyenv/versions/my-environment/lib/python3.7/site-packages/
This needs to point at the packages folder where the installed packages are stored.
If i would want to use multiple, they are seperated by ":"
PYTHONPATH=/home/user/.pyenv/versions/my-environment/lib/python3.7/site-packages/:/home/user/.pyenv/versions/second-env/..

Attention: the suggestoin from @tm-schwartz did not completly work for me, since the " caused some sideeffects. Therefore, in the .env, paths must be without ".

2. Using .dir-locals.el
   In the .dir-locals.el file in the project root, add
   ((python-mode . ((lsp-python-ms-extra-paths . ["/home/user/.pyenv/versions/my-env/lib/python3.7/site-packages/"]))))
   if you want to add multiple paths, the syntax would be ["/path/for/environment1" "/path/for/environment2"]

[vmiheer]

For anyone following #96 (comment), make sure to use absolute path. Don't use ~.