The rdml importer is an open source Wolfram package that allows the validation and import of the standard RDML files generated by most of the widely used qPCR equipments.
This package makes qPCR data, and all of its experimental settings, readily available to Mathematica users without the additional burden of becoming familiarized with the RDML schema and having to learn new package-specific functions.
The files from this repository you will need to have a working package are:
- Package:
rdml.m
; - XML schema:
RDML_v1_2_REC.xsd
, from RDML schema version 1.2; - Documentation notebook:
RDML_doc.nb
.
Example dataset files that accompany this package can be found in the folder ./datasets:
1507AA03.rdml
, downloaded from the RDML Consortium database;QPCRCourseApril2015_plate_1_.rdml
, from Ruijter et al., 2015 but downloaded from HFRC, AMC, the Netherlands;rdml_data.xml
, resultant file from unzippingQPCRCourseApril2015_plate_1_.rdml
;rpa.rdml
, courtesy of Raquel P. Andrade Lab, CBMR, Portugal.
Make sure that rdml.m
is in your current working directory (if not, specify the path).
Then, to load the package, simply run:
Get["rdml.m"]
Once you have loaded the package you may access the documentation notebook via the function:
HelpPageRDML[]
Now, the builtin Import
function should be overloaded to work with .rdml
and .rdm
files.
Try importing one of the example data files (found in the datasets folder). For example rpa.rdml
:
rdml_data = Import["datasets/rpa.rdml"]
The data becomes then available as a Dataset
object, on which all the recently expanded query functionalities of Mathematica can be be applied. For more information on how to take advantage of the new data query language check Computation with Structured Datasets.
The package is fully compatible with RDML version 1.2, and partially backwards compatible with versions 1.0 and 1.1.
Option | Default value | Description |
---|---|---|
ValidateAgainstXSD | False | whether to validate file against RDML schema before importing |
Compressed | Automatic | whether to assume that the file is a compressed archive |
Dataset | True | whether to return a Dataset structure expression |
The RDML Consortium defined a XML Schema (XSD) for the RDML standard. The importer presented here fully supports currently the most recent version 1.2 (RDML_v1_2_REC.xsd
). By default, the importer will not attempt to validate the file against the XML schema. If the file to be imported is not compliant with version 1.2, it will still attempt to import inasmuch as it is compatible with version 1.2.
However, to check if the file to be imported complies with the XML Schema, set the option "ValidateAgainstXSD"->True
. If the file fails to be validated, a warning is issued and Import
returns $Failed
.
Import["datasets/QPCRCourseApril2015_plate_1_.rdml", "ValidateAgainstXSD" -> True]
Conversely, if no errors are detected, the data is simply returned:
Import["datasets/rpa.rdml", "ValidateAgainstXSD" -> True]
Frequently, RDML files do not fully comply with the XML Schema yet the errors are relatively minor. The importer will try to import as much as it can given the Schema, being flexible where it can.
Import["datasets/QPCRCourseApril2015_plate_1_.rdml", "ValidateAgainstXSD" -> False]
According to the RDML Consortium guidelines, the XML file containing the RDML compliant data should be stored in a file named rdml_data.xml
. This file should be compressed into a pkzip compatible archive. The archive can be freely named, however instead of holding the .zip extension, it should hold the .rdml (preferably) or .rdm extension. In addition, RDML compatible software should be able to read compressed .rdml or .rdm files, as well as uncompressed .xml files.
By default the importer tries to determine if the file is compressed or not (default option "Compressed"->Automatic
) by checking if the first two bytes of the file correspond to the ASCII string "PK"
.
For instance, if you'd want to manually check if some file is a pkzip compatible archive, you could run:
Import["datasets/QPCRCourseApril2015_plate_1_.rdml", {"Byte", {1, 2}}] // FromCharacterCode
that should return PK
.
By default, the importer will automatically determine the file compression status and import accordingly.
The following three Import
calls are all equivalent for a compressed RDML file:
Import["datasets/QPCRCourseApril2015_plate_1_.rdml"]
Import["datasets/QPCRCourseApril2015_plate_1_.rdml", "Compressed" -> Automatic]
Import["datasets/QPCRCourseApril2015_plate_1_.rdml", "Compressed" -> True]
Explicitly forcing the importer to assume that the file is not compressed (when it actually is) will result in $Failed
with XML-related parsing errors:
The importer will run smoothly on an uncompressed file (rdml_data.xml
is QPCRCourseApril2015_plate _ 1_. rdml
uncompressed).
(* All equivalent *)
Import["datasets/rdml_data.xml", "RDML", "Compressed" -> False]
Import["datasets/rdml_data.xml", "RDML", "Compressed" -> Automatic]
Import["datasets/rdml_data.xml", "RDML"]
If the file extension is not .rdml or .rdm, then the file type is mandatory, otherwise Import
will read the input file as XML and return a XMLObject
expression:
Import["datasets/rdml_data.xml"]
By default, the expression returned after import is a Dataset
expression. Setting "Dataset"-> False
returns the underlying RDML data explicitly as a nested structure of associations and lists.
Lets import some RDML data first:
rpa = Import["datasets/rpa.rdml"]
To check top elements in the hierarchy:
Keys[rpa] // Normal
The version
element indicates the RDML Schema version of the file. This element can be easily retrieved from the Dataset
object:
rpa["version"]
or directly extracted when importing:
Import["datasets/rpa.rdml", {"RDML", "version"}]
Since its release, the RDML standard has been revised twice: versions 1.1 and 1.2. Although this importer has been designed to fully support version 1.2, in practice, since version 1.2 Schema specification significantly overlaps with previous versions, this package can import RDML files from those previous versions (to the extent that they overlap with version 1.2).
Importing files from versions other than 1.2 results in a warning, yet most data is often successfully imported.
The element dateMade
indicates the date and time stamp of the creation of the file. The element dateUpdated
indicates the date and time stamp of the last update of the file.
rpa["dateMade"]
rpa["dateUpdated"]
Use the id
element to show all ids. Each id can be used to assign a publisher and a serial number to the RDML file. Additionally, an MD5Hash can also be included.
rpa["id"]
The experimenter
element contains a list of researchers and their info (in this case only one):
rpa["experimenter"]
Internally this is represented as an Association
. The key in this Association
corresponds to the XML id attribute from the original RDML file, which can be used by other elements to link here:
To retrieve all information related to one particular experimenter, one can either use the key (XML id) or simply pick out the corresponding part:
The documentation element constitutes the multi-purpose documentation system from the RDML file. This element is a text field with an unique id (translated to a key in an Association
).
From many places in the RDML file, a reference can be made to these documentation elements, making it versatile and allowing the free annotation of the elements.
To retrieve all documentation
elements:
rpa["documentation"]
In this file there is only one such element, whose contents pertain to the genome assembly version used to retrieve the genomic information on the HoxB gene cluster:
rpa["documentation", 1]
Whose text subelement contains:
rpa["documentation", 1, "text"]
"The genomic information for the HoxB cluster is based on the Gallus gallus (chick) genome assembly version 2.1 (WASHUC2), as performed by the Genome Sequencing Center (https://genome.wustl.edu) at the Washington University School of Medicine, St. Louis."
The dye
element contains information regarding the fluorescent chemical compounds used as dyes for the real-time monitoring of the qPCR reaction.
To show all dyes:
The sample element contains all samples used. It may describe standard samples used in a dilution series, but it will often describe different biological sources and/or conditions/treatments performed on biological material.
To inspect the different samples:
rpa["sample"]
Check information associated with one particular sample, e.g. "R-MoH9"
:
rpa["sample", "R-MoH9"]
Each sample can have a type: unkn (unknown sample), ntc (non template control), nac (no amplification control), std (standard sample), ntp (no target present), nrt (minusRT, negative for Reverse Transcription), pos (positive control) or opt (optical calibrator sample).
rpa["sample", All, "type"]
Inspect the description of all samples:
rpa["sample", All, "description"]
The target
element contains all targets. A target is defined by the primer (and probe) mix added to the sample to specifically amplify the target sequence (amplicon).
rpa["target"]
Examine all amplicon sequences:
rpa["target", All, "sequences", "amplicon", "sequence"]
Inspect the forward and reverse primers:
rpa["target", All, "sequences", {"forwardPrimer", "reversePrimer"}, "sequence"] // Dataset
The xRef
subelement relates the targets to an external database:
rpa["target", All, "xRef", 1]
The thermalCyclingConditions
element describes the temperature and time steps taken by a thermocycler in order to amplify the DNA. It can be used to describe alternative cycling programs (e.g., a regular PCR or a cDNA synthesis program).
Show the temperature protocol description:
rpa["thermalCyclingConditions", 1, "description"]
Examine the various steps that compose the actual program:
rpa["thermalCyclingConditions", 1, "step"]
The experiment
element contains the data collected during one (or more) qPCR runs, where each run contains one or more qPCR reactions. Each reaction incorporates one sample (which is referred to by its id) and one or more data elements (one for each target (also referred by its id)). Multiple data elements are only required for a multiplex reaction, in which several targets are simultaneously measured (using different fluorescent dyes). If the reactions are measured in different runs, the data should be stored as separate runs.
Check the first run
in the experiment
element:
rpa["experiment", 1, "run", 1]
This run holds various details, including specifics about the hardware (instrument element) as well as details about the software that collected the data (dataColletionSoftware
); documentation
, experimenter
and thermalCyclingConditions
elements should have id references to elements previously defined. The backgroundDeterminationMethod
and cqDetectionMethod
elements include relevant information about the mathematical analysis of the qPCR amplification curves. The pcrFormat
element allows downstream analysis software to display the data according to the qPCR instrument run format. Finally, the react
element contains the actual data pertaining to the qPCR reactions, containing a list of qPCR reactions, each consisting of two subelements: sample
and data
.
rpa["experiment", 1, "run", 1, "react"]
Examine the summary of cq
values for all reactions of sample "R-MoH9"
:
The adp
and mdp
subelements contain the amplification and melting data, respectively.
Plot amplification curves for all reactions of sample "R-MoH9"
:
amplificationData["R-MoH9"] = rpa["experiment", 1, "run", 1, "react",
Select[#sample == "R-MoH9" &], "data", 1, "adp", All, {"cyc", "fluor"}];
ListPlot[amplificationData["R-MoH9"], Joined -> True, Frame -> True, PlotLabel -> "Amplification curves of R-MoH9 reactions",
FrameLabel -> {"Temperature (Celsius degrees)",
"Fluorescence (a.u)"}]
Plot melting curves for all reactions of sample "R-MoH9"
:
meltingData["R-MoH9"] = rpa["experiment", 1, "run", 1, "react", Select[#sample == "R-MoH9" &], "data", 1, "mdp", All, {"tmp", "fluor"}];
To highlight the melting temperature, calculate the negative of the derivative of the fluorescence with respect to temperature:
meltingData2["R-MoH9"] = Values /@ meltingData["R-MoH9"] // Normal // Values;
interpolatedMeltingData["R-MoH9"] = Interpolation /@ meltingData2["R-MoH9"];
interpolatedDerMeltingData["R-MoH9"] = (-D[#[t], t]) & /@ interpolatedMeltingData["R-MoH9"];
Plot[interpolatedDerMeltingData["R-MoH9"], {t, 65, 95}, PlotRange -> All, Frame -> True, FrameLabel -> {"Temperature (Celsius degrees)", "-dfluor/dT"}]