setproctitle fails if you have imported numpy

[leandropls]

The module seems to fail to find environ address after importing numpy.

Without numpy:

$ SPT_DEBUG=1 python3 -c 'from setproctitle import setproctitle'
[SPT]: module init
[SPT]: reading argc/argv from Python main
[SPT]: found 3 arguments
[SPT]: walking from environ to look for the arguments
[SPT]: found environ at 0x7fff5bfffa31
[SPT]: found argv[2] at 0x7fff5bfffa0b: from setproctitle import setproctitle
[SPT]: found argv[1] at 0x7fff5bfffa08: -c
[SPT]: argv[0] should be at 0x7fff5bfffa00
[SPT]: found argv[0]: python3
$

With numpy:

$ SPT_DEBUG=1 python3 -c 'import numpy; from setproctitle import setproctitle'
[SPT]: module init
[SPT]: reading argc/argv from Python main
[SPT]: found 3 arguments
[SPT]: walking from environ to look for the arguments
[SPT]: found environ at 0x101c80050
[SPT]: zero 2 not found
[SPT]: couldn't find argv from environ
[SPT]: get_argc_argv failed
[SPT]: failed to initialize module setproctitle
$ 

[dvarrazzo]

Uhm... it works for me:

$ SPT_DEBUG=1 python3 -c "import numpy; from setproctitle import setproctitle"
[SPT]: module init
[SPT]: reading argc/argv from Python main
[SPT]: found 3 arguments
[SPT]: walking from environ to look for the arguments
[SPT]: found environ at 0x7ffdcd2c4298
[SPT]: found argv[2] at 0x7ffdcd2c4264: import numpy; from setproctitle import setproctitle
[SPT]: found argv[1] at 0x7ffdcd2c4261: -c
[SPT]: argv[0] should be at 0x7ffdcd2c4259
[SPT]: found argv[0]: python3

My versions are:
Python 3.4.0
setproctitle 1.1.9
numpy 1.8.2

[leandropls]

Mine are:
Mac OS X 10.10.4
Python 3.4.2
setproctitle 1.1.9
numpy 1.9.2

And:
Debian GNU/Linux 8 (jessie) x86_64
Python 3.4.2
setproctitle 1.1.9
numpy 1.9.2

(both fail)

[dvarrazzo]

It would be interesting to know if anything changed in numpy 1.9

We can have a proof by patching tests/pyrun.c this way:

--- a/tests/pyrun.c
+++ b/tests/pyrun.c
@@ -11,18 +11,25 @@
  */

 #include <Python.h>
+#include <stdio.h>
+
+extern char **environ;

 int
 main(int argc, char *argv[])
 {
     int rv = 0;

+    printf("environ before: %p\n", environ);
+
     Py_Initialize();

     if (0 != PyRun_SimpleFile(stdin, "stdin")) {
         rv = 1;
     }

+    printf("environ after:  %p\n", environ);
+
     Py_Finalize();

     return rv;

Compiling it with:

make PYTHON=python3 PYCONFIG=python3-config tests/pyrun3

[Fixed: was make PYTHON=python3 tests/pyrun3]

And running it with:

echo "import numpy" | tests/pyrun3

My result with numpy 1.8.2 is that the environment is not changed:

environ before: 0x7ffc9d37e0d8
environ after:  0x7ffc9d37e0d8

Could you please try this test and report the result? Thank you.

[dvarrazzo]

Sorry, there's a problem and it seems pyrun3 embeds Python 2... sigh. Let me check...

[dvarrazzo]

Ok, from the above, compile pyrun3 with:

make PYTHON=python3 PYCONFIG=python3-config tests/pyrun3

to test it's compiled right:

echo "import sys; print(sys.version_info)" | tests/pyrun3 
environ before: 0x7fffc85fd0a8
sys.version_info(major=3, minor=4, micro=0, releaselevel='final', serial=0)
environ after:  0x7fffc85fd0a8

As you can see on my system the environ isn't moved.

[leandropls]

I can confirm it breaks for numpy>=1.9.0; but yeah, it didn't show changes in environ.

[dvarrazzo]

Can I see the output of:

echo "import sys, numpy; print(sys.version_info)" | tests/pyrun3 

thank you

[leandropls]

Here it is, thank you for your help:

$ echo "import sys, numpy; print(sys.version_info)" | tests/pyrun3 
environ before: 0x7fff5685a918
sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0)
environ after:  0x7fff5685a918

[dvarrazzo]

It is embedding Python2, see my comment above. Delete tests/pyrun3 and rebuild it with make PYTHON=python3 PYCONFIG=python3-config tests/pyrun3.

[leandropls]

Done it:

$ echo "import sys, numpy; print(sys.version_info)" | tests/pyrun3 
environ before: 0x7fff54138918
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
environ after:  0x7f87a9e52040

[dvarrazzo]

This seems to prove that numpy does something to the environ variable.

Unfortunately I don't think I can do anything about it: in Python3 the argv array is not available because Python manipulates it to encode to Unicode. The only way I've found to retrieve argv is to look for it before environ. If someone moves environ too I'm at a loss :(

You should try importing setproctitle before numpy: this should fix it but then I don't know if anything would break in numpy. If this is the case you can set SPT_NOENV=1 and setproctitle will leave environ untouched (but the length of the title will be limited to the argv length).

[leandropls]

Thank you for the effort @dvarrazzo. I wonder why numpy needs to do this... I was looking into their code but couldn't find it yet. The workaround of importing setproctitle before numpy works alright. Thanks!

[dvarrazzo]

No problem. I will close this issue as I don't think I can do much but I'll report the thing to numpy.

Before doing that, can you verify if python3 + numpy 1.8 doesn't change environ? Maybe it's only a Mac OS X thing (I'm on Ubuntu 14.04 here).

echo "import numpy, sys; print('numpy', numpy.__version__); print('Python', sys.version)" | tests/pyrun3
environ before: 0x7ffeae9416a8
numpy 1.8.2
Python 3.4.0 (default, Jun 19 2015, 14:24:19) 
[GCC 4.8.2]
environ after:  0x7ffeae9416a8

[leandropls]

I don't think the bug is OS X related, as I was able to reproduce it on Debian Jessie too.

Python3 + numpy 1.8.2 doesn't change environ on my Mac:

$ echo "import sys, numpy; print(sys.version_info)" | tests/pyrun3 
environ before: 0x7fff502d8918
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
environ after:  0x7fff502d8918

Maybe we could include this as a known issue in the README?

[dvarrazzo]

We know now it's related to numpy 1.9 and not to the OS. Time to hear from them. Maybe they can fix it in 1.9.next.

[dvarrazzo]

So, it seems numpy doesn't do anything but using setenv, and actually this is enough to move the environ pointer. In this case importing setproctitle before should be enough and everything downstream should work as expected.