SpaRC (refer to paper and sources) is capable of forming accurate clusters in terms of both purity and completeness on the long reads. However, it suffers from the under clustering issue on the illumina short reads. In order to solve this issue, we leverage the long connections of the long reads to help clustering the short reads. We optimized original LPA for clustering metagenomic long and short reads. In the HybridLPA, we only allow the long reads labels to propogate throughout the read graph. This site provides MPI/UPC++ based implementation of HybridLPA.

First clone the code

    git clone https://github.com/JGI-Bioinformatics/HybridLPA.git
    cd HybridLPA && git submodule update --init --recursive

The two versions are independent from each other, you may choose to only build the version that you are interested in.

• Linux
• mpi >= 3.0
• c++ standard >= 11

    mkdir build && cd build
    cmake -DBUILD_SPARC_MPI=ON ..
    make sparc_mpi

• Linux
• mpi >= 3.0
• c++ standard >= 11
• upcxx 2020.3.2

Install UPC++ is beyond this scope. Set environment UPCXX_INSTALL helps cmake to find upcxx. Also remember set UPCXX_NETWORK to your network, otherwise smp is most likely be used.

    mkdir build && cd build
    cmake -DBUILD_SPARC_UPCXX=ON ..
    make sparc_upcxx

The HybridLPA is designed for partitioning the read graph. So a read graph is needed to be constructed before using the HybridLPA. There are two ways to construct the read graph: 1) Using [SpaRC] (https://bitbucket.org/LizhenShi/sparc/src/master/README.md); 2) To make the running environment consistent, we also incoporate the steps used to form read graph in SpaRC in this repository.

Forming clusters using HybridLPA on the mixture of long and short reads. In this step there are two options: PowerGraph LPA or lpav1_mpi/upcxx.

lpav1_mpi/upcxx is our implementation of LPA in MPI or UPC++, which is build in the corresponding targets.

PowerGraph(https://github.com/jegonzal/PowerGraph) also provide LPA program which can be got from github.

Remark: Since this step is purely graph clustering, actually any graph clustering algorithms can be used providing it can work on graphs of billions edges.

Find the kmer counting profile of the data, so that we decides how to filter out "bad" kmers. In this step we run edge_generating_$SURFIX where $SURFIX means mrmpi, mimir, mpi or upcxx.

For example for mpi version:

    $./kmer_counting_mpi -h
    
    -h, --help
    shows this help message
    -i, --input
    input folder which contains read sequences
    -p, --port
    port number
    -z, --zip
    zip output files
    -k, --kmer-length
    length of kmer
    -o, --output
    output folder
    --without-canonical-kmer
    do not use canonical kmer

Find shared reads for kmers with kmer_read_mapping_$SURFIX where $SURFIX means mrmpi, mimir, mpi or upcxx.

For example for mpi version:

    $./kmer_read_mapping_mpi -h
    
    -h, --help
    shows this help message
    -i, --input
    input folder which contains read sequences
    -p, --port
    port number
    -z, --zip
    zip output files
    -k, --kmer-length
    length of kmer
    -o, --output
    output folder
    --without-canonical-kmer
    do not use canonical kmer

Generate graph edges using edge_generating_$SURFIX where $SURFIX means mrmpi, mimir, mpi or upcxx.

For example for mpi version:

    $./edge_generating_mpi -h
    
    -h, --help
    shows this help message
    -i, --input
    input folder which contains read sequences
    -p, --port
    port number
    -z, --zip
    zip output files
    -o, --output
    output folder
    --max-degree
    max_degree of a node; max_degree should be greater than 1
    --min-shared-kmers
    minimum number of kmers that two reads share. (note: this option does not work)
    

Please find sbatch scripts of sample runs on LAWRENCIUM in misc/example folder.

When running the programs, make sure there are enough nodes to hold all the data in memory. Although some programs support storing temporary data in disk, but it will make the progress quite slow.