docker-compose installation fails

[guillaume-be]

Hello,

I am trying to install Opus-MT via the docker-compose solution, but I am encountering an error during the compilation step:

Marian compilation error

In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:24,
                 from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15:
/usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h: In instantiation of 'T* sentencepiece::model::FreeList<T>::Allocate() [with T = sentencepiece::unigram::Lattice::Node]':
/usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:83:41:   required from here
/usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h:62:13: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct sentencepiece::unigram::Lattice::Node'; use assignment or value-initialization instead [-Wclass-memaccess]
       memset(chunk, 0, sizeof(*chunk) * chunk_size_);
       ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15:
/usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:38:10: note: 'struct sentencepiece::unigram::Lattice::Node' declared here
   struct Node {
          ^~~~
In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:24,
                 from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15:
/usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h: In instantiation of 'void sentencepiece::model::FreeList<T>::Free() [with T = sentencepiece::unigram::Lattice::Node]':
/usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:93:24:   required from here
/usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h:39:13: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct sentencepiece::unigram::Lattice::Node'; use assignment or value-initialization instead [-Wclass-memaccess]
       memset(chunk, 0, sizeof(*chunk) * chunk_size_);
       ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15:
/usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:38:10: note: 'struct sentencepiece::unigram::Lattice::Node' declared here
   struct Node {
          ^~~~

Python package installation error

  ----------------------------------------
  Failed building wheel for tornado
  Running setup.py clean for tornado
  Running setup.py bdist_wheel for mosestokenizer: started
  Running setup.py bdist_wheel for mosestokenizer: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/mosestokenizer/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-thswx0b2 --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for mosestokenizer
  Running setup.py clean for mosestokenizer
  Running setup.py bdist_wheel for pycld2: started
  Running setup.py bdist_wheel for pycld2: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/pycld2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-v5m0lqhx --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for pycld2
  Running setup.py clean for pycld2
  Running setup.py bdist_wheel for sqlitedict: started
  Running setup.py bdist_wheel for sqlitedict: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/sqlitedict/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-8em57psk --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for sqlitedict
  Running setup.py clean for sqlitedict
  Running setup.py bdist_wheel for docopt: started
  Running setup.py bdist_wheel for docopt: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/docopt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-1536sh2e --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for docopt
  Running setup.py clean for docopt
  Running setup.py bdist_wheel for toolwrapper: started
  Running setup.py bdist_wheel for toolwrapper: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/toolwrapper/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-svf3gk7k --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for toolwrapper
  Running setup.py clean for toolwrapper
  Running setup.py bdist_wheel for uctools: started
  Running setup.py bdist_wheel for uctools: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/uctools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-i2p1v3j0 --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for uctools
  Running setup.py clean for uctools
Failed to build tornado mosestokenizer pycld2 sqlitedict docopt toolwrapper uctools
Installing collected packages: tornado, docopt, openfile, toolwrapper, uctools, mosestokenizer, pycld2, sqlitedict, sentencepiece
  Running setup.py install for tornado: started
    Running setup.py install for tornado: finished with status 'done'
  Running setup.py install for docopt: started
    Running setup.py install for docopt: finished with status 'done'
  Running setup.py install for toolwrapper: started
    Running setup.py install for toolwrapper: finished with status 'done'
  Running setup.py install for uctools: started
    Running setup.py install for uctools: finished with status 'done'
  Running setup.py install for mosestokenizer: started
    Running setup.py install for mosestokenizer: finished with status 'done'
  Running setup.py install for pycld2: started
    Running setup.py install for pycld2: finished with status 'done'
  Running setup.py install for sqlitedict: started
    Running setup.py install for sqlitedict: finished with status 'done'

Runtime error

λ docker-compose up
Creating network "opus-mt_default" with the default driver
Starting opus-mt_opus-mt_1 ... done
Attaching to opus-mt_opus-mt_1
opus-mt_1  | marian-server: error while loading shared libraries: libsentencepiece_train.so.0: cannot open shared object file: No such file or directory
opus-mt_1  | ERROR:asyncio:Future exception was never retrieved
opus-mt_1  | future: <Future finished exception=CalledProcessError(127, 'unknown')>
opus-mt_1  | Traceback (most recent call last):
opus-mt_1  |   File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 742, in run
opus-mt_1  |     yielded = self.gen.throw(*exc_info)  # type: ignore
opus-mt_1  |   File "server.py", line 49, in run
opus-mt_1  |     ret = yield self.p.wait_for_exit()
opus-mt_1  |   File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 735, in run
opus-mt_1  |     value = future.result()
opus-mt_1  | subprocess.CalledProcessError: Command 'unknown' returned non-zero exit status 127.

I am running Windows 10 with Docker on a WSL2 backend. Is there any pre-requisite step in order to build the container?
Thank you,

[thecarmo]

Hey,

I'm also trying to run the dockerized version of the service and just had the same issues. I believe there are some problems in the Dockerfile which I tried to solve as follows:

• It seems that the python-slim base image removes gcc in order to maintain its size Unable to execute 'gcc': No such file or directory docker-library/python#60 -> I just reversed to normal base image, that solved the "unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1" error. You'll get a bigger image but it will work

• The python packages in the requirements.txt files were correctly installed in the virtual environment venv but the python interpreter used to run the server was the default one, so using https://pythonspeed.com/articles/activate-virtualenv-dockerfile/ I've reordered and rewritten some of the commands. Also wheel needed to be installed prior to requirements.txt so that "Using legacy 'setup.py install' for mosestokenizer, since package 'wheel' is not installed." and similar issues are solved

• Finally, after downloading the models and adding these modifications I got the service running. Remember to acces the container IP which might not be localhost https://stackoverflow.com/questions/17157721/how-to-get-a-docker-containers-ip-address-from-the-host

Code a little less organized than what's on the elg branch but it got the service running.
Dockerfile.txt

Thanks @mhepaixao and @Aguiar16 for the discussion on this issue.
Rafael