md5checksum shows example dataset analysis fails #13

RishiDeKayne · 2021-03-16T14:50:00Z

Hi, I've been trying to use dentist on the provided example dataset but a number of the md5 check sums after it finishes running are failing with no other errors that I can find.

I installed snakemake v6.0.0 and singularity v3.6.3 through conda and ran through the example dataset as follows:

wget https://bds.mpi-cbg.de/hillerlab/DENTIST/dentist-example.v1.0.1.tar.gz
tar -xzf ./dentist-example.v1.0.1.tar.gz
cd dentist-example

# run the workflow
SKIP_LACHECK=1 snakemake --configfile=snakemake.yaml --use-singularity --cores=4 

# validate the files
md5sum -c checksum.md5

but the checksum output was as follows:

gap-closed.fasta: FAILED
workdir/.assembly-test.bps: OK
workdir/.assembly-test.dentist-reads.anno: OK
workdir/.assembly-test.dentist-reads.data: OK
workdir/.assembly-test.dentist-self.anno: OK
workdir/.assembly-test.dentist-self.data: OK
workdir/.assembly-test.dust.anno: OK
workdir/.assembly-test.dust.data: OK
workdir/.assembly-test.hdr: OK
workdir/.assembly-test.idx: OK
workdir/.assembly-test.tan.anno: OK
workdir/.assembly-test.tan.data: OK
workdir/.gap-closed-preliminary.bps: FAILED
workdir/.gap-closed-preliminary.dentist-self.anno: FAILED
workdir/.gap-closed-preliminary.dentist-self.data: FAILED
workdir/.gap-closed-preliminary.dentist-weak-coverage.anno: FAILED
workdir/.gap-closed-preliminary.dentist-weak-coverage.data: FAILED
workdir/.gap-closed-preliminary.dust.anno: FAILED
workdir/.gap-closed-preliminary.dust.data: FAILED
workdir/.gap-closed-preliminary.hdr: OK
workdir/.gap-closed-preliminary.idx: FAILED
workdir/.gap-closed-preliminary.tan.anno: FAILED
workdir/.gap-closed-preliminary.tan.data: FAILED
workdir/.reads.bps: OK
workdir/.reads.idx: OK
workdir/assembly-test.assembly-test.las: OK
workdir/assembly-test.dam: OK
workdir/assembly-test.reads.las: OK
workdir/gap-closed-preliminary.dam: FAILED
workdir/gap-closed-preliminary.fasta: FAILED
workdir/gap-closed-preliminary.gap-closed-preliminary.las: FAILED
workdir/gap-closed-preliminary.reads.las: FAILED
workdir/reads.db: OK
md5sum: WARNING: 15 computed checksums did NOT match

any advice on how to get the example dataset running would be greatly appreciated,
Thanks,
Rishi

The text was updated successfully, but these errors were encountered:

a-ludi · 2021-03-16T20:20:32Z

Hi Rishi, could you share one of the logs/process.*.log files? Somebody else experienced failing md5sums like you do and the reason was that one of the auxiliary tools crashed in most of the calls for a yet unknown reason. Could you also share some more information about your system?

lsb_release -a
free -h

RishiDeKayne · 2021-03-16T20:51:56Z

Sure, I have attached process.1.log and the system info is as follows:

lsb_release -a

output:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

free -h

output:

              total        used        free      shared  buff/cache   available
Mem:           995G        1.8G        229G        324K        764G        988G
Swap:          8.0G        1.5G        6.5G

process.1.log

a-ludi · 2021-03-17T08:35:44Z

As I suspected, it is the same error memory-associated error:

$ jq 'select((.exitStatus // 0) != 0)' process.1.log | head -n50
{
  "thread": 140513968151344,
  "logLevel": "diagnostic",
  "state": "post",
  "command": [
    "computeintrinsicqv",
    "-d19",
    "/tmp/dentist-processPileUps-OeaddP/pileup-55b-56f.db",
    "/tmp/dentist-processPileUps-OeaddP/pileup-55b-56f.pileup-55b-56f-chained-filtered.las"
  ],
  "output": [
    "allocation failure: Invalid argument cachelinesize=0 requested size is 24",
    "AutoArray<unsigned long,alloc_type_memalign_cacheline> failed to allocate 3 elements (24 bytes)",
    "current total allocation 467987",
    "",
    ""
  ],
  "exitStatus": 1,
  "timestamp": 637514242850290800,
  "action": "execute",
  "type": "command"
}
... (many more instances with the same signature)

The problem is clearly not related to a lack of memory. Since I have no in-depth knowledge of computeintrinsicqv, I will ask the author for help.

In the meantime, you may try running it on a different machine.

a-ludi · 2021-03-17T08:52:37Z

Information from other user:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          1.0Ti        15Gi       2.5Gi       4.1Gi       989Gi       982Gi
Swap:            0B          0B          0B
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

RishiDeKayne · 2021-03-19T12:04:51Z

Hi again,
Weirdly I reran the example set each of our computing nodes - it failed on every one of our big memory machines but ran on our regular machines. I did the same system checks as above but cant find anything obviously different between the two so I'm still not sure what could be causing it. In case it is helpful:

$ lsb_release -a

##WORKED - regular 
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

##FAILED - big-memory 
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

$ free -h

##WORKED - regular
              total        used        free      shared  buff/cache   available
Mem:           360G        5.2G        332G        4.4M         21G        354G
Swap:          8.0G        8.0G         88K

##FAILED - big memory  
              total        used        free      shared  buff/cache   available
Mem:           995G        1.8G        229G        324K        764G        988G
Swap:          8.0G        1.5G        6.5G

and now all checksum outputs say 'OK'

a-ludi · 2021-03-22T11:34:51Z

Hmm, interesting. I will try running the example on a 1TB memory machine as well. Maybe there is some bug related to large pointers.

shri1984 · 2021-04-06T06:38:34Z

Hi,
I have the same issue. md5sum -c checksum.md5 failed (15 cases). I am using a machine with 2 TB RAM (Ubuntu).

a-ludi · 2021-04-19T07:53:23Z

I tried it on one of our big memory machines and it worked as expected:

# submit job with 8 cores
$ sbatch -c8 -pbigmem --wrap='snakemake --configfile=snakemake.yaml --use-singularity --cores=$SLURM_JOB_CPUS_PER_NODE'
# memory information about the machine
$ ssh r01n03 free -h
              total        used        free      shared  buff/cache   available
Mem:           1.0T        964G         40G        1.6G        2.5G         39G
Swap:            0B          0B          0B
# OS information about the machine
$ ssh r01n03 lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:la
nguages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.4.1708 (Core) 
Release:        7.4.1708
Codename:       Core

So I conjecture (:smile:) that it is not just the amount of total or available memory that causes the bug. But I still have no clue what's going on. Also, I have not heard anything from the author of daccord (see this issue). I will keep digging.

a-ludi · 2021-06-22T12:02:21Z

@shri1984 @RishiDeKayne I hope you are still interested in DENTIST after all this time but I think I have fixed the bug (25f96d2). I would be very happy if you could test the example again and see if it works.

The issue (likely) was that I used Alpine Linux in the Container which has its own libc implementation that is not 100% compatible with glibc used in common Distros like Ubuntu. I switched to Ubuntu and the error went away on one of my machines.

shri1984 · 2021-08-11T19:56:27Z

Thanks @a-ludi. example data set went fine including the md5sum. The latest version helped. I am trying dentist on my hic scaffolded hifi assembly. I will post the update here.

lizhao007 · 2022-11-26T09:18:31Z

@shri1984 @RishiDeKayne I hope you are still interested in DENTIST after all this time but I think I have fixed the bug (25f96d2). I would be very happy if you could test the example again and see if it works.

The issue (likely) was that I used Alpine Linux in the Container which has its own libc implementation that is not 100% compatible with glibc used in common Distros like Ubuntu. I switched to Ubuntu and the error went away on one of my machines.

Thanks for your work, but I get the same issue with example data by the latest version (v4.0.0) — md5sum -c checksum.md5 failed (15 cases). The information about the machine is:

              total        used        free      shared  buff/cache   available
Mem:           2.0T        535G        1.4T         56M        5.7G        1.4T
Swap:          4.0G        2.1G        1.9G

LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core) 
Release:	7.5.1804
Codename:	Core

a-ludi · 2022-12-05T14:19:28Z

Hi @lizhao007 ,

could you please share the list of files that failed the checksum test? I need it to get an idea what went wrong.

a-ludi self-assigned this Mar 16, 2021

a-ludi added bug Something isn't working help wanted Extra attention is needed labels Mar 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

md5checksum shows example dataset analysis fails #13

md5checksum shows example dataset analysis fails #13

RishiDeKayne commented Mar 16, 2021 •

edited

Loading

a-ludi commented Mar 16, 2021

RishiDeKayne commented Mar 16, 2021

a-ludi commented Mar 17, 2021

a-ludi commented Mar 17, 2021

RishiDeKayne commented Mar 19, 2021

a-ludi commented Mar 22, 2021

shri1984 commented Apr 6, 2021 •

edited

Loading

a-ludi commented Apr 19, 2021

a-ludi commented Jun 22, 2021

shri1984 commented Aug 11, 2021

lizhao007 commented Nov 26, 2022

a-ludi commented Dec 5, 2022

md5checksum shows example dataset analysis fails #13

md5checksum shows example dataset analysis fails #13

Comments

RishiDeKayne commented Mar 16, 2021 • edited Loading

a-ludi commented Mar 16, 2021

RishiDeKayne commented Mar 16, 2021

a-ludi commented Mar 17, 2021

a-ludi commented Mar 17, 2021

RishiDeKayne commented Mar 19, 2021

a-ludi commented Mar 22, 2021

shri1984 commented Apr 6, 2021 • edited Loading

a-ludi commented Apr 19, 2021

a-ludi commented Jun 22, 2021

shri1984 commented Aug 11, 2021

lizhao007 commented Nov 26, 2022

a-ludi commented Dec 5, 2022

RishiDeKayne commented Mar 16, 2021 •

edited

Loading

shri1984 commented Apr 6, 2021 •

edited

Loading