PIP Installation issue on some linux / docker environment

[raychew13]

I've been using fbprophet in my local dev environment (Mac) for a while for forecasting purpose and It's AWESOME.

I'm planning to run my forecasting engine on EC2. But I have issue with installing the fbprophet. I've also managed to reproduce the installation issue on my docker container. Below is the command that I used to test the installation of the fbprophet library in the runtime docker container.
docker run --rm -i -t python:3.6.0 bash -c "pip install fbprophet==0.2"

I'm using python3.6 docker image from docker-hub. Just wondering if there is any issue with the specific version of gcc. I've also checked the version of gcc installed in python3.6 docker image. It has gcc (Debian 4.9.2-10) 4.9.2

Below is the error.

Installing collected packages: pyparsing, six, cycler, numpy, python-dateutil, pytz, matplotlib, pandas, Cython, pystan, fbprophet
  Running setup.py install for fbprophet ... error
    Complete output from command /usr/local/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6027ivd3/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-khm0g75a-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_models
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1fdd477dd8bcc50395c4e25c8741f1c1 NOW.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/local/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6027ivd3/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-khm0g75a-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-6027ivd3/fbprophet/

[bletham]

I'm not certain what the issue is but one issue that has come up before in VMs is that it actually requires a lot of RAM to compile the models. Does this instance have >2GB of RAM?

[raychew13]

Hi Ben,
Thanks for the response.
The issue is in the installation of fbprophet library in the docker container. If you are using docker in your local machine, you can easily reproduce the issue by running the below command.

docker run --rm -i -t python:3.6.0 bash -c "pip install fbprophet==0.2"

Please let me know if you have additional information.

Regards,
Ray

[ghost]

I have installed fbprophet on numerous memory sizes. It requires 4GB to compile - and I got the tokenize error during Pystan. It wouldn't even install on a C4 Large (3.75GB), but it did work at 4GB. To install on EC2, I used a T2.Medium, and then downgraded to actually run it. This does apply in your case, because per documentation [1], there are no resource constraints initially placed on the docker VMs, which could very well mean that is never getting the 4 GB required to compile Pystan. I have confirmed on EC2 using CENT OS images that it will not work on a C4 Large (3.75GB), but will work on a 4GB, using the exact same instance - merely changing instance type. By default, Docker for MAC is set to a maximum of 2GB per VM. [2] You may have hit this situation on yours. It will run fine with 2GB, but the installation will fail to compile with less than 4GB, and will fail even at 3.75GB on C4.Large on AWS. [1] https://docs.docker.com/engine/admin/resource_constraints/#limit-a-containers-access-to-memory [2] https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container

…

On Thu, Oct 12, 2017 at 11:44 PM, Ray Chew ***@***.***> wrote: Hi Ben, Thanks for the response. The issue is in the installation of fbprophet library in the docker container. If you are using docker in your local machine, you can easily reproduce the issue by running the below command. docker run --rm -i -t python:3.6.0 bash -c "pip install fbprophet==0.2" Please let me know if you have additional information. Regards, Ray — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZTbFqh00iwu2ESkalvaSA03TLrjWTMMks5srtyJgaJpZM4PzaW6> .

[bletham]

@robertdknight Thanks for the details on the RAM requirements. I updated the description in the docs to include the 4GB install requirement (d1bf4a1).

[dtourillon]

Based on the error traces I see in the initial issue's description, I seem to have the same problem when trying to install fbprophet on my laptop with 16G of RAM.

@raychew13 : was your problem related to RAM?

@bletham : any other idea as to what could be causing the problem, apart from insufficient RAM?

See #346 for the ticket I've created for my issue, before finding this one.

[CarstVaartjes]

Hi,

I have a similar issue, but doing this on a local docker image of amazonlinux:2017.03 where my laptop has ubuntu 17.10 and 32gb, so I'm hesitant to think that it's a memory issue.

Building wheels for collected packages: fbprophet
  Running setup.py bdist_wheel for fbprophet ... error
  Complete output from command /sklearn_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zXYmeu/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpljkJYmpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/fbprophet
  creating build/lib/fbprophet/stan_models
  INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1fdd477dd8bcc50395c4e25c8741f1c1 NOW.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for fbprophet
  Running setup.py clean for fbprophet
Failed to build fbprophet
Installing collected packages: fbprophet
  Running setup.py install for fbprophet ... error
    Complete output from command /sklearn_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zXYmeu/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-s6Sfs4-record/install-record.txt --single-version-externally-managed --compile --install-headers /sklearn_build/include/site/python2.7/fbprophet:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_models
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1fdd477dd8bcc50395c4e25c8741f1c1 NOW.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/sklearn_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zXYmeu/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-s6Sfs4-record/install-record.txt --single-version-externally-managed --compile --install-headers /sklearn_build/include/site/python2.7/fbprophet" failed with error code 1 in /tmp/pip-build-zXYmeu/fbprophet/

From what I understand, docker doesn't limit the memory usage of a container in Linux, so what could it be then?

Edit: I just saw #346 and tried the example code there.

>>> model = pystan.StanModel(model_code=model_code)
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_5944b02c79788fa0db5b3a93728ca2bf NOW.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sklearn_build/lib64/python2.7/site-packages/pystan/model.py", line 313, in __init__
    build_extension.run()
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 448, in build_extensions
    self.build_extension(ext)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 498, in build_extension
    depends=ext.depends)
  File "/usr/lib64/python2.7/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib64/python2.7/distutils/unixccompiler.py", line 134, in _compile
    raise CompileError, msg
distutils.errors.CompileError: command 'gcc' failed with exit status 1

So the issue is pystan here and not fbprophet, but i do have gcc installed: Package gcc-4.8.5-1.22.amzn1.noarch already installed and latest version

Will check for known pystan issues with docker/amazon linux

See also: stan-dev/pystan#219
Going back to cython 0.22 from 0.27.3 did not help. So it must be something else...

It seems to be a gcc issue such as here: stan-dev/pystan#395

edit: I solved it by uninstall gcc and then:

yum install gcc64
yum install gcc64-c++

After that everything was in sync; apparently pystan doesn't like gcc<5 and I ended up with GCC 4 & 6 at the same time which confused cython amongst others.

[bletham]

Great to hear a solution, thanks! It looks like we should add to the linux installation instructions to check that gcc is 6 or higher?

[ghost]

I am using GCC 4.8.5 and it works on many different systems, so that 6 or higher may be related to a specific distribution. If this is Ubuntu, you could check the configuration mentioned at https://docs.docker.com/engine/installation/linux/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities, because debian distributions do not support that out of the box. You are running an RPM distribution in a docker container (which does support memory auditing) on an OS that doesn't, but could be made that way via configurations. Shuhei Kagawa has a great investigation at https://shuheikagawa.com/blog/2017/05/27/memory-usage/ into memory usage of Docker containers with Alpine Linux and the investigation as to why the image cannot see the same memory details as the host system (if the resources are truly unconstrained). Fabio Kung wrote a detailed investigation for Heroku on memory in docker containers and why free and top do not work, and that is at https://fabiokung.com/2014/03/13/memory-inside-linux-containers/. My suspicion is that the docker container is unable to see 4GB+ of Ram in whatever inquiry mode is used by the installation script on that particular OS.

…

On Fri, Nov 17, 2017 at 11:59 AM, Ben Letham ***@***.***> wrote: Great to hear a solution, thanks! It looks like we should add to the linux installation instructions to check that gcc is 6 or higher? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZTbFhejzVj8Bp99qAmD2TXuRPFgyvZEks5s3cmagaJpZM4PzaW6> .

[bletham]

I updated the install instructions for Red Hat to include gcc64, hopefully that helps others but I'm going to go ahead and close this issue and just point others with Linux install issues here in the future.

[Gypsying]

I met the same problems when i pip install fbprophet:
CompileError: command 'gcc' failed with exit status 1

my env :
centos 7.2
Python 3.6
numpy 1.16.2
cython 0.29.7
pystan 2.19.0.0

I change the pystan to 2.18.1.0 solve the problems.

[bletham]

hm I'm not sure what might be happening there. I just installed pystan 2.19 and pip install fbprophet went smoothly after that.
I wonder if maybe something went wrong in the pystan 2.19 installation. I'm glad it's working now, but if you are interested in further debugging I'd be interested to know if the following code runs in pystan 2.19:

import pystan
model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
model = pystan.StanModel(model_code=model_code)  # this will take a minute
y = model.sampling(n_jobs=1).extract()['y']
y.mean()  # should be close to 0

[bletham]

Actually it looks like the integration tests on Travis are now failing for the same reason, and was also fixed by setting pystan<=2.18.1. I'm opening an issue for it in #930