Skip to content
This repository has been archived by the owner on May 5, 2020. It is now read-only.

Error when run pipeline with docker file #28

Open
Ritzttk opened this issue Sep 24, 2018 · 4 comments
Open

Error when run pipeline with docker file #28

Ritzttk opened this issue Sep 24, 2018 · 4 comments
Labels
bug Something isn't working

Comments

@Ritzttk
Copy link

Ritzttk commented Sep 24, 2018 •
edited by ewels
Loading

Hi, when I ran this command :

nextflow run nf-core/vipr -params-file vi.yaml -profile docker

I met this error:

N E X T F L O W  ~  version 0.31.1
Launching `nf-core/vipr` [gloomy_almeida] - revision: 1ca3412b9d [master]
==================================================
 nf-core/vipr : Viral amplicon/enrichment analysis and intrahost variant calling: v1.0-dev
==================================================
List of samples: PDH203_GTGGCC
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$trim_and_combine = <value>` with a process selector
[warm up] executor > local
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$decont = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$kraken = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$tadpole = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$gap_fill_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$polish_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$final_mapping = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$var_calling = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$genomecov = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$vipr_tools = <value>` with a process selector
[c6/04f746] Submitted process > trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)
ERROR ~ Error executing process > 'trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)'

Caused by:
  Process `trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)` terminated with an error exit status (125)

Command executed:

  # loop over readunits in pairs per sample
  pairno=0
  echo 425-N706-S517_S6_L001_R1_001.fastq 425-N706-S517_S6_L001_R2_001.fastq | xargs -n2 | while read fq1 fq2; do
      let pairno=pairno+1
      # note: don't make reads smaller than assembler kmer length
      skewer --quiet -t 4 -m pe -q 3 -n -l 31 -z -o pair${pairno}-skewer-out $fq1 $fq2;
      cat *-trimmed-pair1.fastq.gz >> PDH203_GTGGCC_R1-trimmed.fastq.gz;
      cat *-trimmed-pair2.fastq.gz >> PDH203_GTGGCC_R2-trimmed.fastq.gz;
      rm *-trimmed-pair[12].fastq.gz;
  done
  fastqc -t {task.cpus} PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz;

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'nfcore/vipr:1.0-dev' locally
  docker: Error response from daemon: manifest for nfcore/vipr:1.0-dev not found.
  See 'docker run --help'.

Work dir:
  /home/vi/work/c6/04f746274f2e2baec8dcb1b457ba47

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details
Oops... Pipeline execution stopped with the following message: Unable to find image 'nfcore/vipr:1.0-dev' locally
docker: Error response from daemon: manifest for nfcore/vipr:1.0-dev not found.
See 'docker run --help'.
Pipeline execution summary
    ---------------------------
    Completed at : Mon Sep 24 03:55:39 UTC 2018
    Duration     : 5.6s
    Success      : false
    Work Dir     : /home/vi/work
    Exit status  : 125
    Error report : Error executing process > 'trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)'

Caused by:
  Process `trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)` terminated with an error exit status (125)

Command executed:

  # loop over readunits in pairs per sample
  pairno=0
  echo 425-N706-S517_S6_L001_R1_001.fastq 425-N706-S517_S6_L001_R2_001.fastq | xargs -n2 | while read fq1 fq2; do
      let pairno=pairno+1
      # note: don't make reads smaller than assembler kmer length
      skewer --quiet -t 4 -m pe -q 3 -n -l 31 -z -o pair${pairno}-skewer-out $fq1 $fq2;
      cat *-trimmed-pair1.fastq.gz >> PDH203_GTGGCC_R1-trimmed.fastq.gz;
      cat *-trimmed-pair2.fastq.gz >> PDH203_GTGGCC_R2-trimmed.fastq.gz;
      rm *-trimmed-pair[12].fastq.gz;
  done
  fastqc -t {task.cpus} PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz;

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'nfcore/vipr:1.0-dev' locally
  docker: Error response from daemon: manifest for nfcore/vipr:1.0-dev not found.
  See 'docker run --help'.

Work dir:
  /home/vi/work/c6/04f746274f2e2baec8dcb1b457ba47

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

WARN: To render the execution DAG in the required format it is required to install Graphviz -- See https://www.graphviz.org for more info.

After that, I think I have prolem with the url of docker, So I edited the nextflow.config from
container = "nfcore/vipr:${version}" // for GIS translated into conda env name!
to
container = "nfcore/vipr
So the pipeline worked on first process. But it error on the second process

>N E X T F L O W  ~  version 0.31.1
Launching `nf-core/vipr` [ridiculous_boyd] - revision: 1ca3412b9d [master]
==================================================
 nf-core/vipr : Viral amplicon/enrichment analysis and intrahost variant calling: v1.0-dev
==================================================
List of samples: PDH203_GTGGCC
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$trim_and_combine = <value>` with a process selector
[warm up] executor > local
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$decont = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$kraken = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$tadpole = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$gap_fill_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$polish_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$final_mapping = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$var_calling = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$genomecov = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$vipr_tools = <value>` with a process selector
[f8/88c7c1] Submitted process > trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)
[42/27852b] Submitted process > decont (Decontaminating PDH203_GTGGCC)
ERROR ~ Error executing process > 'decont (Decontaminating PDH203_GTGGCC)'

Caused by:
  Process `decont (Decontaminating PDH203_GTGGCC)` terminated with an error exit status (1)

Command executed:

  decont.py -i PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz -t 4 -c 0.5 -r human_g1k_v37.fasta -o PDH203_GTGGCC_trimmed_decont;
  # since this is the last fastqc processing step, let's run fastqc here
  fastqc -t {task.cpus} PDH203_GTGGCC_trimmed_decont_1.fastq.gz PDH203_GTGGCC_trimmed_decont_2.fastq.gz;

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  Traceback (most recent call last):
    File "/opt/conda/bin/decont.py", line 449, in <module>
      assert shutil.which('bwa') and shutil.which('samtools')
  AttributeError: 'module' object has no attribute 'which'

Work dir:
  /home/vi/.nextflow/assets/nf-core/vipr/work/42/27852be75b8444b072ea29c2bbaa61

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
Oops... Pipeline execution stopped with the following message: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/opt/conda/bin/decont.py", line 449, in <module>
    assert shutil.which('bwa') and shutil.which('samtools')
AttributeError: 'module' object has no attribute 'which'
Pipeline execution summary
    ---------------------------
    Completed at : Mon Sep 24 04:09:36 UTC 2018
    Duration     : 24.8s
    Success      : false
    Work Dir     : /home/vi/.nextflow/assets/nf-core/vipr/work
    Exit status  : 1
    Error report : Error executing process > 'decont (Decontaminating PDH203_GTGGCC)'

Caused by:
  Process `decont (Decontaminating PDH203_GTGGCC)` terminated with an error exit status (1)

Command executed:

  decont.py -i PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz -t 4 -c 0.5 -r human_g1k_v37.fasta -o PDH203_GTGGCC_trimmed_decont;
  # since this is the last fastqc processing step, let's run fastqc here
  fastqc -t {task.cpus} PDH203_GTGGCC_trimmed_decont_1.fastq.gz PDH203_GTGGCC_trimmed_decont_2.fastq.gz;

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  Traceback (most recent call last):
    File "/opt/conda/bin/decont.py", line 449, in <module>
      assert shutil.which('bwa') and shutil.which('samtools')
  AttributeError: 'module' object has no attribute 'which'

Work dir:
  /home/vi/.nextflow/assets/nf-core/vipr/work/42/27852be75b8444b072ea29c2bbaa61

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

WARN: To render the execution DAG in the required format it is required to install Graphviz -- See https://www.graphviz.org for more info.

I think, i had problem with the docker file. Do you recommend I will use docker or using conda for software dependencies ?.

Kiên
@ewels ewels added the bug Something isn't working label Sep 24, 2018
@ewels ewels mentioned this issue Sep 24, 2018
Merged
@ewels
Copy link
Member

ewels commented Sep 24, 2018

Hi @Ritzttk,

You have two problems here. The first you correctly diagnosed and fixed, is the missing docker image tag. This is resolved in #30 by essentially doing what you did above - removing the :1.0-dev prefix (though I replace it with :latest, which is the default).

After this, you have a warning that I've not seen before:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

Though this doesn't seem to be a blocking problem. Maybe @pditommaso has seen this before?

Then you also have a python error from the decont.py script:

Traceback (most recent call last):
  File "/opt/conda/bin/decont.py", line 449, in <module>
    assert shutil.which('bwa') and shutil.which('samtools')
AttributeError: 'module' object has no attribute 'which'

A quick bit of googling suggests that this is because the version of Python is too old. I've just specified Python 3.6 in the conda environment file (also in #30), hopefully this will solve the problem.

Phil

@pditommaso
Copy link

pditommaso commented Sep 24, 2018 •
edited
Loading

Never seen, but googling for it, it may be a problem with your docker version or the base image.

@ewels
Copy link
Member

ewels commented Sep 25, 2018

Hi @Ritzttk,

The new Docker hub image is now building. Hopefully when it's done the AttributeError problem should be fixed, but we'll see. Please let me know how you get on!

Cheers,

Phil

@Ritzttk
Copy link
Author

Ritzttk commented Sep 26, 2018

Hi @ewels ,
The pipeline with new Docker image works now. Thank you and your team so much.

Kiên

andreas-wilm pushed a commit to andreas-wilm/vipr that referenced this issue Feb 7, 2019
Fixed testing setup and many minor nf-core compat issues (lint warnings)
a402a0e 
 - Added test profile and activated in travis.yaml
 - Added name to license (closes nf-core#25)
 - Fixed container version tagging (closes nf-core#28)
 - Upgraded Nextflow version dep
 - deleted test-data dir (now in nf-core testdata repo)
 - Fixed all nf-core lint problems (closes nf-core#10)
 - Contamination index now mandatory (closes nf-core#8)
 - Upgraded most programs (env.yaml)
 - Enabled --reads input (rather than just via params-file)

 Please enter the commit message for your changes. Lines starting
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ewels @pditommaso @Ritzttk