[WIP] WORK IN PROGRESS AND NOT YET STABLE - DO NOT USE FOR PRODUCTIVE SETTINGS YET
nf-core/datasync is a system operation pipeline that provides several workflows for handling system operation / automation tasks that are commonly helpful for various tasks in large data processing / analysis facilities. This includes:
- Data Synchronization & Checksum generation
- Configurable: Can provide YAML file which files to include or exclude from sync
- Checksum backend: Can configure which backend to use for checksum generation (e.g. sha256sum, md5, ...)
- Configurable whether to include (sub-) folders in the sync or not (search for checkpoint files, e.g. has to have DEMUX_DONE that signals a demultiplexing run was finished & successfully copied)
- Data Integrity validation
- Provided with a directory to check, can validate that file(s) found are matching checksums from Synchronization subworkflow
- Data Archival & Deletion
- Can check source and target location for existence of file(s) and decide based on user configurable rules whether files can be considered archived
- Timestamp older than X days
- Checksums match Integrity validation report
- Create empty files to make it obvious that archival was performed
- Optionally: Delete files or create list of files to be deleted for manual deletion process
- Can check source and target location for existence of file(s) and decide based on user configurable rules whether files can be considered archived
The pipeline can be configured by users to execute any of the aforementioned subworkflows and then produces a report using MultiQC custom content that also serves as a report of what was done by the pipeline for documentation purposes.
Note
If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.
Now, you can run the pipeline using:
nextflow run nf-core/datasync \
-profile <docker/singularity/.../institute> \
--input samplesheet.csv \
--outdir <OUTDIR>
--sync
--sync_backend 'sha256'
--sync_done true #Creates SYNC_DONE file when done in each folderWarning
Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters;
see docs.
For more details and further functionality, please refer to the usage documentation and the parameter documentation.
To see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page. For more details about the output files and reports, please refer to the output documentation.
nf-core/datasync was originally written by Alexander Peltzer.
We thank the following people for their extensive assistance in the development of this pipeline:
If you would like to contribute to this pipeline, please see the contributing guidelines.
For further information or help, don't hesitate to get in touch on the Slack #datasync channel (you can join with this invite).
An extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.
You can cite the nf-core publication as follows:
The nf-core framework for community-curated bioinformatics pipelines.
Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.
Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.