Downloader script sometimes does not exit properly (hangs) after a download fails

[dougli1sqrd]

The Downloader script scripts/download_source_gafs.py in The Pipeline in Jenkins the script has hung when attempting to exit after one of the downloaded files failed.

The downloader is supposed to, in parallel, download multiple GAF/etc sources for all (or some subset) of groups, and if a dataset fails to download, retry some number of times. If that dataset still fails after the retries then the downloader should exit.

Instead, in this bug, after the retries we can see that the downloader is attempting to quite, as it announces Aborting! in the logs. But Jenkins is clearly still waiting for the downloader to exit.

Log output from Jenkins here:

00:02:50  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig_rna-src.gaf.gz
00:02:50  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig-src.gaf.gz
00:02:51  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig_isoform-src.gaf.gz
00:03:01  Downloading ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.sgn.gz
00:03:01  Download of ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.sgn.gz failed: wget: unable to resolve host address ‘ftp.solgenomics.net’
00:03:01   - Trying again
00:03:18  Downloading ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.sgn.gz
00:03:31  Download of ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.sgn.gz failed: wget: unable to resolve host address ‘ftp.solgenomics.net’
00:03:31   - Trying again
00:03:49  Download failed! */var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/sgn-src.gaf.gz*
00:03:49  Aborting!

Tagging @kltm

[kltm]

Last instance I have was: 2020-01-31

00:02:22  Download of ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.interpro.sgn.gz failed: No such file ‘gene_association.interpro.sgn.gz’.
00:02:22
00:02:22   - Trying again
00:02:23  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig_complex-src.gaf.gz
00:02:23  Downloading ftp:https://ftp.ebi.ac.uk/pub/databases/GO/goa/UNIPROT/goa_uniprot_all.gaf.gz
00:02:23  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig-src.gaf.gz
00:02:23  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig_rna-src.gaf.gz
00:02:24  Downloaded successfully to /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_pig_isoform-src.gaf.gz
00:02:39  Downloading ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.interpro.sgn.gz
00:02:39  Download of ftp:https://ftp.solgenomics.net/ontology/GO/gene_association.interpro.sgn.gz failed: No such file ‘gene_association.interpro.sgn.gz’.
00:02:39
00:02:39   - Trying again
00:02:53  Download failed! */var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/sgn-src.gaf.gz*

Quoting me from the gitter:
"The downloader didn't seem to be able to bail properly on failure. It might be good to test it against bad or unresolvable URLs."

[kltm]

Another instance today.

[kltm]

Another instance yesterday.

[dougli1sqrd]

Another instance today

00:47:36  Download of ftp:https://ftp.ebi.ac.uk/pub/databases/GO/goa/CHICKEN/goa_chicken_complex.gaf.gz failed: Error in server response. Closing.
00:47:36  Error in server response. Closing.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36   - Trying again
00:47:36  Download of ftp:https://ftp.ebi.ac.uk/pub/databases/GO/goa/CHICKEN/goa_chicken_isoform.gaf.gz failed: Error in server response. Closing.
00:47:36  Error in server response. Closing.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36  failed: Connection refused.
00:47:36   - Trying again
00:47:46  Downloading ftp:https://ftp.ebi.ac.uk/pub/databases/GO/goa/COW/goa_cow.gaf.gz
00:47:46  Download failed! */var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/dictybase-src.gaf.gz*
00:47:46  Aborting!

[kltm]

Instance today with:

00:23:18  failed: Connection refused.
00:23:18  failed: Connection refused.
00:23:18  failed: Connection refused.
00:23:18  failed: Connection refused.
00:23:18   - Trying again
00:23:24  Download of ftp:https://ftp.ebi.ac.uk/pub/databases/GO/goa/DOG/goa_dog.gaf.gz failed: failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24  failed: Connection refused.
00:23:24   - Trying again
00:23:34  Download failed! */var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/goa_cow_rna-src.gaf.gz*
00:23:34  Downloading ftp:https://ftp.ebi.ac.uk/pub/databases/GO/goa/DOG/goa_dog_complex.gaf.gz
00:23:34  Aborting!

[kltm]

Instance on:

00:03:10  Downloading https://www.arabidopsis.org/download_files/GO_and_PO_Annotations/Gene_Ontology_Annotations/gene_association.tair.gz
00:03:10  Download of https://www.arabidopsis.org/download_files/GO_and_PO_Annotations/Gene_Ontology_Annotations/gene_association.tair.gz failed: wget: unable to resolve host address 'www.arabidopsis.org'
00:03:10   - Trying again
00:03:25  Downloading https://www.arabidopsis.org/download_files/GO_and_PO_Annotations/Gene_Ontology_Annotations/gene_association.tair.gz
00:03:43  Download of https://www.arabidopsis.org/download_files/GO_and_PO_Annotations/Gene_Ontology_Annotations/gene_association.tair.gz failed: wget: unable to resolve host address 'www.arabidopsis.org'
00:03:43   - Trying again
00:03:58  Download failed! */var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/tair-src.gaf.gz*
00:03:58  Aborting!
18:33:38  Sending interrupt signal to process
Aborted by kltm
18:33:44  Terminated
18:33:44  script returned exit code 143

[kltm]

Instance on: Build #1513 (Jun 10, 2021 12:00:00 AM)

00:02:39  
00:02:39   - Trying again
00:02:57  Downloading ftp:https://ftp.wormbase.org/pub/wormbase/species/c_elegans/PRJNA13758/annotation/go_annotations/c_elegans.PRJNA13758.current.go_annotations.gaf.gz
00:02:57  Download of ftp:https://ftp.wormbase.org/pub/wormbase/species/c_elegans/PRJNA13758/annotation/go_annotations/c_elegans.PRJNA13758.current.go_annotations.gaf.gz failed: No such file ‘c_elegans.PRJNA13758.current.go_annotations.gaf.gz’.
00:02:57  
00:02:57   - Trying again
00:03:12  Download failed! */var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/target/wb-src.gaf.gz*
00:03:12  Aborting!
11:24:39  Sending interrupt signal to process
11:24:43  Terminated
11:24:43  script returned exit code 143

[kltm]

Noting an annoying instance of this on the 14th, which then may have gummed up the works for the next two attempts over the weekend. Cleaning the workspaces out on the machine and then letting it retry naturally tonight.

[kltm]

It seems that Dec 1 release attempt got caught on this.

[kltm]

Xref geneontology/pipeline#259

[kltm]

Instance on #1814 Oct 20

[kltm]

Instance on Nov 1 (release).