Bio::SeqAlignment::Examples::EnhancingEdlib - Parallelizing Edlib with MCE, OpenMP using Inline and FFI::Platypus
version 0.03
A collection of examples that demonstrate how to enhance the sequence alignment library Edlib by adding process level parallelism (through MCE) and thread level parallelism (through OpenMP). These examples hopefully demonstrate the general present day relevance of Perl for constructing bioinformatic applications related to sequence mapping.
This distribution provides examples of sequence mapping applications build on top of the Edlib library. The examples demonstrate how to use the MCE and OpenMP parallelism libraries to speed up the sequence alignment process through process and thread level parallelism. The examples are meant to be illustrative of the capabilities of Perl to create parallel components for sequencing applications. These examples educational, and are not meant to be used in production code. However, they can be used as a starting point for creating such applications, which is the point we are trying to make in the presentation given at the Perl and Raku conference 2024.
All scripts were used to generate data for the talk given to the Science Track of the Perl & Raku conference 2024 https://tprc.us/tprc-2024-las/ https://blogs.perl.org/users/oodler\_577/2024/01/perl-raku-conference-2024-to-host-a-science-track.html
This is a directory that holds the sequence data used in the examples. It contains two files : Homo_sapiens.GRCh38.cds.sample.strip.fasta : 1000 sequences Homo_sapiens.GRCh38.sample.end.strip.fasta : 648 sequences from the human transcriptome.
This is a directory that holds various scripts in Perl and R that are used to generate and analyze performance data for the MCE and openMP enhanced versions of edlib.
Perl script that illustrates the use of the Linear dataflow and the Generic
Mapper dicussed in Bio::SeqAlignment::Components::SeqMapping
for the creation of an MCE and OpenMP enhanced version of Edlib. Running the
script generates the results file timings_openMP.txt . Before running in your
own machine, please make sure that the number of workers and the number of
threads are set between 1 and the maximum number of hyperthreads for your
system.
Perl script that illustrates the use of the Linear dataflow and the Generic
Mapper dicussed in Bio::SeqAlignment::Components::SeqMapping
for the creation of an MCE enhanced version of Edlib. Running the
script generates the results file timings.txt . Before running in your
own machine, please make sure that the number of workers is set between 1
and the maximum number of hyperthreads for your system. Feel free to
experiment with different chunk sizes (the script only checks chunk sizes
between 1 and 20 query sequences).
R script that generates the figures for the benchmarking between the MCE and MCE/OpenMP enhanced versions of Edlib.
Execution timings of the MCE version of the enhanced Edlib library. This is a tab separated file with three columns: Workers : number of MCE workers Time : execution time in seconds Chunk_Size : MCE chunk size
Execution timings of the MCE/openMP version of the enhanced Edlib library. This is a tab separated file with three columns: Workers : number of MCE workers Time : execution time in seconds Num_Threads : Number of OpenMP threads
Plot of the timings.txt file
Plot of the execution times of the MCE enhanced version of Edlib (for chunk size of one) for an increasing number of MCE workers v.s. the MCE/OpenMP enhancement for an increasing number of threads for single worker.
Plot of the execution times of the MCE/OpenMP enhanced version of Edlib for various combinations of workers and threads. The figure is a two dimensional histogram (rendered as a contour plot) with the 5% best performing combinations marked with a '+' sign.
Plot of the resources (copies of data in memory x execution time) used by
the MCE enhanced version of Edlib (for chunk size of one) for an increasing
number of MCE workers v.s. the MCE/OpenMP enhancement for an increasing
number of threads for single worker.
Plot of the resources (copies of data in memory x execution time) used by the MCE/OpenMP enhanced version of Edlib for various combinations of workers and threads. The figure is a two dimensional histogram (rendered as a contour plot) with the 5% best performing combinations marked with a '+' sign.
Main effects of the number of workers and threads on the execution time of the MCE/OpenMP enhanced version of Edlib. The figure is generated by fitting a Generalized Additive Model (GAM) analysis of the data in the timings_MCE_with_OMP.txt file and plotting the main effects of the number of workers and threads.
Interaction effects of the number of workers and threads on the execution time of the MCE/OpenMP enhanced version of Edlib. The figure is generated by fitting a Generalized Additive Model (GAM) analysis of the data in the timings_MCE_with_OMP.txt file and plotting the generalized interaction term between the number of workers and threads.
-
This distribution provides edlib static and shared libraries so that they can be used by other Perl distributions that are on CPAN.
-
A collection of tools and libraries for aligning biological sequences from within Perl.
-
Bio::SeqAlignment::Components::SeqMapping
Components (modules for dataflow in sequencing applications and generic mappers) to create sequence mapping applications in Perl.
-
Bio::SeqAlignment::Components::Libraries::edlib
Edlib sequence mapping library binding for Perl. Two versions are provided, one that is FFI based and one that is Inline based.
-
MCE (Many-Core Engine) is a high performance parallel processing library for Perl. It is designed to take advantage of the many cores available on modern processors and provides a simple interface for creating parallel components in Perl.
-
The OpenMP ARB (Architecture Review Boards) mission is to standardize directive-based multi-language high-level parallelism that is performant, productive and portable. Jointly defined by a group of major computer hardware and software vendors and major parallel computing user facilities, the OpenMP API is a portable, scalable model that gives shared-memory parallel programmers a simple and flexible interface for developing parallel applications on platforms ranging from embedded systems and accelerator devices to multicore systems and shared-memory systems.
-
The Perl Data Language (PDL) gives standard Perl the ability to compactly store and speedily manipulate the large N-dimensional data arrays which are the bread and butter of scientific computing. PDL turns Perl into a free, array-oriented, numerical language that can be a very solid alternative to switching to Python or R for numerical computations during complex data analysis tasks and pipelines.
Christos Argyropoulos [email protected]
This software is copyright (c) 2024 by Christos Argyropoulos.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.