MIT-licensed; see LICENSE.
- MapReduce and Elastic MapReduce
- Rail-dbGaP protocol specification
- Setting up Amazon Web Services
- Studying k-mers in dbGaP-protected samples with EMR
- A note on TCGA data and other data sources
- Helpful notes for administrators
The National Institutes of Health (NIH) maintains security requirements and recommmendations for analyzing controlled-access genomic data, including dbGaP-protected data. Rail-dbGaP is a protocol described in this preprint for securely analyzing dbGaP-protected genomic data from the Sequence Read Archive (SRA) in the cloud with Amazon Elastic MapReduce (EMR) in a manner compliant with NIH guidelines. The protocol is implemented in Rail-RNA, software for scalable analysis of many hundreds of RNA sequencing (RNA-seq) samples. A step-by-step guide for setting up Rail-RNA to analyze dbGaP-protected RNA-seq data is provided in the Rail documentation here; the present document contains a technical specification of the Rail-dbGaP protocol and walks the user through an example implementation that counts the number of input dbGaP samples on SRA (i.e., SRA run accession numbers) in which each k-mer present in at least one read from among the samples appears. A preprint describing the Rail-dbGaP protocol is available here.
The MapReduce programming model divides a problem into a sequence of alternating computation and aggregation steps. Each step is performed by distributing independent tasks across workers in a cluster of computers. Elastic MapReduce (EMR) is a Hadoop-based implementation of MapReduce especially for a cluster of Elastic Compute Cloud (EC2) instances, or virtualized computers, on Amazon Web Services, a commercial cloud provider. EMR reads input from the web and/or Simple Storage Service (S3), Amazon's cloud storage service, and writes its output back to S3.
Rail-dbGaP secures an EMR cluster so it is compliant with NIH guidelines as follows. (See Figure 1 of the preprint for an illustration. The sections Setting up Amazon Web Services and Studying k-mers in dbGaP-protected samples with EMR put together protocol elements to show their implementation explicitly.)
-
The cluster is created within a subnet of a Virtual Private Cloud (VPC), a logically isolated unit of the cloud providing a private network and firewall. The connection with S3 is via a VPC endpoint, which ensures that data transferred never leaves the data center. Each instance has a public IP. The VPC is set up by creating a stack with the CloudFormation template
cloudformation/dbgap.template
. -
Inbound traffic to the cluster is restricted via security groups. A security group is essentially a stateful firewall. A master security group for the master instance and a worker security group for worker instances prevent initiation of any connection to the cluster except by essential web services. These web services correspond to particular IPs and ports, and the most restrictive sets for master and worker instances are configured automatically. SSH access to the cluster is also restricted: the only interaction between user and cluster is via the EMR interface, which presents progress information through the essential web services. Security groups are also set up by creating a stack with the CloudFormation template
cloudformation/dbgap.template
. Master and worker instances must be associated with security groups when creating the EMR cluster. -
Data are encrypted at rest. During cluster setup, before any sensitive data has reached the cluster, each instance runs the preliminary script (i.e., bootstrap action)
bootstraps/encrypt_local_storage.sh
that uses Linux Unified Key Setup (LUKS) to create an encrypted partition with a keyfile. The key is randomly generated on each instance and never exposed to the user. Temporary files, the Hadoop distributed file system, and buffered output to the cloud storage service are all configured to reside on the encrypted partition via symbolic links. (See line 128 ofbootstraps/encrypt_local_storage.sh
.) Files written to cloud storage are also encrypted; Amazon S3 uses AES256. This is enforced by the creation of a bucket with a policy (i.e., rules governing user access to the bucket) barring uploads that do not turn on server-side encryption in the dbGaP templatecloudformation/dbgap.template
and by setting the EMRFS configuration parameterfs.s3.enableServerSideEncryption=true
. -
Data are encrypted in transit. Worker instances download dbGaP data using SRA Tools, ensuring encryption of data transferred from dbGaP to the cluster. AWS enables Secure Sockets Layer (SSL) by default for transfers between cloud storage and the cluster as well as between cloud storage service and compliant local storage to which an investigator saves results.
-
Identities are managed to enforce the principle of least privilege. The principle of least privilege prescribes users have only the privileges required to perform necessary tasks. In the Rail-dbGaP protocol, an administrator grants the user only those privileges required to run Hadoop programs on EMR clusters. The administrator uses multi-factor authentication and constrains the user to set up a password satisfying NIH's requirements listed among security best practices (minimum of 12 characters, no complete dictionary words, etc.) On AWS, an account administrator configures an Identity and Access Management (IAM) user expressly for running Hadoop jobs and retrieving results from S3, and the password rules described above are enforced. This is achieved as described below in Setting up Amazon Web Services.
-
Audit logs are enabled. These record logins and actions taken by the user and on the user's behalf, including API calls made by processes running on the cluster. On AWS, audit logs take the form of CloudTrail logs stored in encrypted S3 buckets. They are enabled when a stack is created with the dbGaP template
cloudformation/dbgap.template
.
The steps below create a new AWS IAM account especially for analyzing dbGaP-protected data. To perform these steps, both user and AWS site administrator should be available. (For many investigators, user and administrator will be the same person.) It is recommended that they are physically together to minimize passing of credentials. Before continuing, the user should install the AWS Command Line Interface (CLI). Optionally, the user may also have requested access to some dbGaP-protected sample on the Sequence Read Archive (SRA) and received a key file with an ngc
extension.
These steps should be performed if the site administrator is new to AWS.
- Navigate to https://aws.amazon.com/free in your web browser.
- Click Create a free account.
- Check the I am a new user box and and continue to follow the instructions to create your new account. You'll enter, for example, contact and payment information. Note that the Basic level of service is sufficient for our purposes.
- Make a note of your account number.
- Log into the AWS console using the new account's email address and password.
- Click on the arrow next to your user name in the gray banner at the top of the page.
- Select My Account, and the Account Id will be displayed at the top of the page.
- Secure the account
- Log into the AWS console using the new account's email address and password.
- Open the Identity and Access Management page.
During this process, it is best for the account administrator to sit with the user to minimize passing credentials.
-
Administrator: create a new IAM user.
- From the new user's computer, log into the AWS Console and select Identity and Access Management.
- Click Users on the left pane, then Create New Users on the right pane.
-
User: register credentials with the AWS CLI by entering
aws configure --profile dbgap
at a terminal prompt on the user's computer. Enter the AWS Access Key ID, AWS Secret Access Key, and a default region as prompted. We recommend using us-east-1
because its connection to dbGaP-protected data on SRA appears to be fastest. A default output format need not be specified. Now the new user can issue AWS API calls via the AWS CLI. It is recommended that credentials file that was just downloaded is now deleted.
-
Administrator: Set user's password.
- Return to the AWS Console, again click Identity and Access Management, again click Users on the left sidebar, and select the new user. Under User Actions, click Manage Password.
-
User: navigate to the login page URL from credentials (2).csv, log in, and change the password as prompted.
CloudFormation facilitates creation and management of a group of related AWS resources. Rail-dbGaP provides a CloudFormation template for creating a Virtual Private Cloud (VPC) with a single public subnet. An EMR job flow that analyzes dbGaP data should be launched into this subnet. The VPC is supplemented by several security features, including
- a VPC endpoint for S3, which ensures that the connection between the Elastic MapReduce cluster and S3 is private.
- security groups that block all inbound traffic to the cluster from the internet except from the Elastic MapReduce webservice.
- the creation of a secure bucket on S3 into which all output should be written when operating on dbGaP-protected data. The bucket has an attached policy barring uploads that do not have server-side encryption (AES256) turned on.
- CloudTrail logs recording AWS API calls. These are written to the secure bucket.
The administrator should grab the latest version of the template here. Implement it by following these steps. (If the administrator already has CloudTrail turned on, they may not work, causing a rollback. An administrator satisfied with their CloudTrail configuration may instead want to use this alternative CloudFormation template, which creates the VPC but does not attempt to set up CloudTrail.)
- Click CloudFormation in the AWS console, making sure the region in the upper-right corner of the screen is the same as the user's default region (typically
us-east-1
, i.e., N. Virginia).
The best defense is a good offense, and you are encouraged to monitor traffic to clusters launched by the user. You may want to explore turning on VPC flow logs and CloudWatch alarms for suspicious activity.
The new IAM user still needs sufficient privileges to use Elastic MapReduce.
- Return to the AWS Console, again click Identity and Access Management, but now click Policies on the left sidebar.
- Click Create Policy, then select Create Your Own Policy. (You may need to click Get Started first.)
-
Under Policy Name, enter "UseExistingEMRRoles".
-
Under Policy Document, paste the following.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:GetInstanceProfile", "iam:GetRole", "iam:PassRole" ], "Resource": "*" } ] }
-
Click Create Policy.
-
- Now click Users in the left pane, select the new IAM user, and click the Permissions tab.
-
Administrator: follow these instructions to create default roles for Elastic MapReduce.
-
User: run
aws emr create-default-roles --profile dbgap
to retrieve the default Elastic MapReduce roles created by the administrator.
This section reviews the implementation of an EMR pipeline that ingests dbGaP-protected data and counts the number of samples in which each k-mer found in least one read across samples appears. dbGaP support has kindly provided a dataset composed of public RNA-seq samples from 1000 Genomes exclusively for testing secure cloud-based pipelines. Its project accession number on SRA is SRP041052, and the steps below use three samples from it.
Assume the secure bucket created during AWS setup is at s3:https://rail-dbgap-secure
. The following is performed on the user's computer, where the AWS CLI was installed.
-
Download the dbGaP repository key for the test data at
ftp:https://ftp.ncbi.nlm.nih.gov/sra/examples/decrypt_examples/prj_phs710EA_test.ngc
. Upload the key to S3 securely with the AWS CLI by enteringaws s3 cp /path/to/prj_phs710EA_test.ngc s3:https://rail-dbgap-secure/test/prj_phs710EA_test.ngc --profile dbgap --sse
It is recommended that you delete the key from your computer with
rm /path/to/prj_phs710EA_test.ngc
-
Using a text editor, create a script
copy_files_to_node.sh
with the following contents.#!/usr/bin/env bash set -ex aws s3 cp s3:https://rail-dbgap-secure/test/prj_phs710EA_test.ngc /mnt/space/DBGAP.ngc aws s3 cp s3:https://rail-emr/step.sh /mnt/space/step.sh
Copy the script to S3 as follows.
aws s3 cp /path/to/copy_files_to_node.sh s3:https://rail-dbgap-secure/test/ --profile dbgap --sse
This script will be a bootstrap action that (1) securely transfers the key to each EC2 instance and (2) copies the mapper script to each instance. The mapper script is discussed below. 3. Download this list of three SRA run accession numbers from test dbGaP project SRP041052. Copy it to S3 as follows
aws s3 cp /path/to/manifest.txt s3:https://rail-dbgap-secure/test/ --profile dbgap --sse
- Navigate to the login page URL from credentials (2).csv. The AWS console should appear immediately or after you log in with the credentials from this CSV.
- Click EMR, then Create cluster, and finally Go to advanced options.
bash /mnt/space/step.sh
This is the script here, and it is reproduced below.
#!/usr/bin/env bash
set -ex;
cut -f2 | { IFS= read -r SRR;
KMERSIZE=21;
cd /mnt/space/sra_workspace/secure;
fastq-dump ${SRR} --stdout -X 10000 \
| bioawk -v kmersize=${KMERSIZE} -v srr=${SRR} -c fastx \
'{
for (i=1; i<=length($seq)-kmersize; i++) {
revcompsubseq = substr($seq, i, kmersize);
subseq = revcompsubseq;
revcomp(revcompsubseq);
if (revcompsubseq < subseq) {
print "UniqValueCount:" revcompsubseq "\t" srr;
} else {
print "UniqValueCount:" subseq "\t" srr;
}
}
}'
}
It describes a mapper that uses SRA Tools fastq-dump
to grab an input sample from SRA and Heng Li's bioawk to extract k-mers (for k=21) from its read sequences to print either the k-mer or its reverse complement, whichever is first in lexicographic order. UniqValueCount
refers to how aggregation should be performed by Hadoop Streaming: the reducer described in the next step will count the number of unique run accession numbers associated with a given k-mer. The command-line parameter -X 10000
of fastq-dump
grabs only the first 10,000 reads of each sample for the purpose of demonstration only. Both SRA Tools and bioawk will have to be installed by bootstrap scripts, which are configured in later steps.
9. Next to Reducer, enter
aggregate
This allows the UniqValueCount
s output by the mappers to be interpreted properly. See this page for more information on the Hadoop Aggregate package.
10. Next to Input S3 location, enter
s3:https://rail-dbgap-secure/test/manifest.txt
A given line of this file, which you uploaded to S3 in an earlier step, is passed to each mapper. 11. Next to Output S3 location, enter
s3:https://rail-dbgap-secure/test/out/
This is where the number of samples in which each k-mer appears will be written. 12. In the Arguments box, enter
-inputformat org.apache.hadoop.mapred.lib.NLineInputFormat
to tell Hadoop that each mapper should be passed a single line of the input file manifest.txt
. The box in which you're entering data should now look as depicted below.
While we do not provide explicit instruction on how to download TCGA data, the user may substitute the SRA Tools bootstrap and fastq-dump
for analogs that use CGHub's GeneTorrent or the Seven Bridges API. Similar substitutions apply to other protected data sources. No other parts of the Rail-dbGaP protocol need modification.
As for any new AWS account, you should consider how you would like to configure billing. Consolidated billing can be convenient if you are managing multiple AWS accounts simultaneously.
Further, you should consider raising your EC2 instance limits. This is particularly important if you plan to analyze large datasets (more than 100 RNA-seq samples at a time). To raise your limits, visit this page.