We have developed an Amazon Machine Image (AMI) that is optimized for running Bioconductor in the Amazon Elastic Compute Cloud (or EC2) for sequencing tasks.
Here are a few reasons you could use it:
See below for more specific scenarios.
The AMI comes pre-loaded with the latest release version of R and the top 80 Bioconductor software packages plus the following categories of annotation packages:
First you will need an Amazon Web Services (AWS) account if you do not already have one. Sign up for AWS and then click here to sign up for the EC2 service. This will require that you provide credit card information; however, you will only be charged for services used. Some AWS services are free.
That’s all that is required if you want to use RStudio Server to connect to your AMI with a web browser. If you also want to connect to it with SSH, create a keypair as follows:
Launch the AWS Console. Click on the Key Pairs link in the lower left-hand corner of the page. Click the “Create Key Pair” button. When prompted, supply a name. We suggest that the name be a combination of “bioconductor”, your first name, and your machine name (this will avoid conflicts with other people who share your AWS account, or possibly your own account on another machine). For example, if your name is Bob and your personal computer is named “mylaptop”, your key pair name could be “bioconductor-bob-mylaptop”. Download the resulting .pem file and keep it in a safe place.
To launch the AMI you’ll step through several screens:
View the progress of the launching instance by going to the EC2 dashboard then click ‘running instances’. Once the instance is ‘running’ it’s ready to go.
Important Note: When you are finished using the AMI, be sure to stop or terminate the instance. Click on the instance and select the desired action from the ‘Actions’ -> ‘Instance State’. If you stop the instance you are still charged for storage. Termination eliminates all further charges but also looses any changes you’ve made and any new session needs to be started fresh from one of the pre-loaded AMIs.
Start one of the pre-loaded AMIs.
Follow the same steps as above. On the ‘Running Instances’ page select the instance and get the public IP from the ‘Description’ tab at the bottom.
If the public IP was 220.127.116.11 the basic ssh command is
ssh -i bioconductor-bob-mylaptop.pem firstname.lastname@example.org
You can vary this command. If you want to use programs or R packages that use X11, be sure and add a -X flag, to make the command something like this:
ssh -X -i bioconductor-bob-mylaptop.pem email@example.com
Now you can paste your command line into a terminal or Command Prompt. Make sure you are in the same directory as your key pair file.
Windows Users: You will need to install a version of the ssh and scp commands. Graphical programs like PuTTY and WinSCP will work. Our examples, however, will use the command-line versions of these programs, which you can obtain by installing Cygwin (be sure and install the openssh package).
Once you have pasted this command into your Terminal or Command Prompt window (and pressed Enter) you should be connected to your Amazon EC2 instance.
Each instance that receives a public IP address is also given an external DNS hostname. The Public IP is part of the hostname, for example if the Public IP is 50-16-120-30, the external DNS hostname is ec2-50-16-120-30.compute-1.amazonaws.com. Note the use of dash in the DNS hostname vs dots in the IP.
Paste the hostname in your browser and it should take you to the RStudio
Server login page. Log in with the username
ubuntu and password
Our AMIs have the following IDs.
|Bioconductor Version||R Version||AMI ID|
|3.4 (release, recommended)||3.3.2||ami-e43e1af3|
Please note that AMI IDs may change over time as we update the underlying AMI. Refer to this page for the most current AMI IDs. These AMIs live in the US-East-1 region.
For administrative and funding reasons, Bioconductor keeps track of each time a Bioconductor AMI is launched. No identifying information is kept. By using the AMI, you consent to this tracking.
Make sure you have connected to your instance either with a web browser, or using the -X flag of the ssh command. Something like:
ssh -X -i bioconductor-bob-mylaptop.pem firstname.lastname@example.org
Then, from within R on the remote instance:
library("Rgraphviz") set.seed(123) V <- letters[1:10] M <- 1:4 g1 <- randomGraph(V, M, 0.2) plot(g1)
This should start a graphics device on your local computer displaying a simple graph.
This works best if you have selected a high-CPU instance type to run.
This trivial example runs the
rnorm() function, but any function would work. Consult the
parallel documentation for more information.
library(parallel) mclapply(1:30, rnorm)
You can also use the AMI as a cluster, wherein the machines communicate with each other via one of the following mechanisms:
In order to use the Bioconductor AMI in one of these scenarios, you need to install StarCluster, which is a software package designed to automate and simplify the process of building, configuring, and managing clusters of virtual machines on Amazon’s EC2 cloud.
StarCluster takes care of the details of making sure that the machines can communicate with each other, including:
Note: Using the Bioconductor AMI for cluster-related tasks is only supported for the Bioconductor AMI version 2.14 and higher.
Install StarCluster by following the StarCluster Installation Guide. This is a simple and fast process.
If you are on Windows, go directly to the Windows section.
Before we can use StarCluster with the Bioconductor AMI,
we need to configure it, by editing its
You can create the file by issuing the command:
This will give you three options:
Options: --------  Show the StarCluster config template  Write config template to /home/user/.starcluster/config [q] Quit
Choose option 2 and note the location of the config file (it will be different from what is shown above).
On Unix systems (including Linux and Mac OS X), this file is found
~/.starcluster/config. On Windows systems, the
folder should be located in your
config file in your favorite text editor, and edit it
You need to fill in values for
AWS_SECRET_ACCESS_KEY. If you don’t know these values,
go to the Security Credentials
page of the AWS Console and expand the “Access Keys” section.
You can view or create your access keys here. Be sure and
store these credentials in a safe place (in addition to
your StarCluster config file).
The value of
AWS_USER_ID can also be found on
the Security Credentials page, by expanding
the “Account Identifiers” section. Fill in
with the number shown as your “AWS Account ID” (this
should be a 12-digit number with hyphens).
If you haven’t already created a keypair in EC2, please do so now by reading the keypairs section.
You can also create a keypair with StarCluster; run the command
starcluster createkey --help
Remember the name that you assigned to your keypair. Change the line
mykey is replaced by the name you assigned
to your keypair in EC2, and change the following line
So that the value of
KEY_LOCATION is the full path to
the private key downloaded from .ec2 (it probably has a
StarCluster allows you to define multiple clusters in the config
file. For now let’s just modify the cluster defined as
KEYNAMEto the name of your key pair (see keypair section above).
CLUSTER_SIZEto the number of machines you want to launch. This number includes the master node, so the default value of 2 means one master, and one worker. We recommend starting with 2 until you have familiarized yourself with using StarCluster and Bioconductor.
DNS_PREFIX = True. This makes your cluster instances easier to recognize when using the AWS Console or command line tools.
NODE_IMAGE_IDto the AMI-ID of the AMI you want to use This will be listed in the AMI IDs section of this document. Note that StarCluster only works with AMIs for Bioconductor version 2.14 and higher.
NODE_INSTANCE_TYPEto another instance type. See the Instance Types page for more information.
#PERMISSIONS = ssh, http, add the line
permissions = http(note lowercase). This is related to security group permissions (more about this below).
You can make additional changes to this section if you want to further customize your configuration. Refer to the StarCluster documentation for more information.
Remove the comments (
# symbol) from the four lines
so that you end up with:
[permission http] IP_PROTOCOL = tcp FROM_PORT = 80 TO_PORT = 80
This allows port 80 on the cluster instances to be open to the world, allowing us to use Rstudio Server on that port.
Assuming you have completed the steps above, you can create a cluster with the command:
starcluster start smallcluster
After a few moments, the cluster should be available.
There are two ways to connect to the cluster’s master node: RStudio Server and SSH. Unless you have a special need to use SSH, we recommend using RStudio Server.
First, get the hostname of the master node by issuing the command:
You can also abbreviate this:
This will produce output like the following:
----------------------------------------------- smallcluster (security group: @sc-smallcluster) ----------------------------------------------- Launch time: 2014-06-16 09:57:54 Uptime: 0 days, 02:19:56 Zone: us-east-1b Keypair: bioc-default EBS volumes: N/A Cluster nodes: smallcluster-master running i-46a76c6d ec2-54-91-23-93.compute-1.amazonaws.com smallcluster-node001 running i-47a76c6c ec2-54-224-6-153.compute-1.amazonaws.com Total nodes: 2
The line that starts with
smallcluster-master ends with a hostname
(in this case it’s
ec2-54-91-23-93.compute-1.amazonaws.com; in your
case it will be something different but similar). You can paste
this host name into a web browser (depending on the browser, you may
need to put
http:// in front of the host name).
This should bring you to the RStudio Server login page. You can log
in with the username
ubuntu and the password
To connect to the master node using ssh, simply issue the command
starcluster sshmaster --user=ubuntu smallcluster
When you are done, you MUST terminate your cluster or you will continue to be charged money by Amazon Web Services. To terminate the cluster, do this:
starcluster terminate smallcluster
This command will prompt you to confirm that you really want to terminate the cluster.
The following scenarios assume that you have started up a cluster and that you are connected to the master node.
When you start a cluster with StarCluster, it’s automatically
configured to use the
with Sun Grid Engine as the back end. You can demonstrate this
by loading BatchJobs:
Among other output, this will say
cluster functions: SGE
indicating that Sun Grid Engine is the back end.
Here’s how to send a simple job to the cluster:
library(BatchJobs) library(BiocParallel) param <- BatchJobsParam(2, resources=list(ncpus=1)) register(param) FUN <- function(i) system("hostname", intern=TRUE) xx <- bplapply(1:100, FUN) table(unlist(xx))
This will produce:
smallcluster-master smallcluster-node001 50 50
…indicating that SGE ran half the jobs on the master and the other half on the worker node.
This presentation outlines how to develop a full-scale analysis (identification of cis-dsQTL) using the kind of cluster we’ve just created.
Here is the same example as above, except using SSH instead of Sun Grid Engine as the back end:
library(BatchJobs) library(BiocParallel) cluster.functions <- makeClusterFunctionsSSH( makeSSHWorker(nodename="smallcluster-master"), makeSSHWorker(nodename="smallcluster-node001") ) param2 <- BatchJobsParam(2, resources=list(ncpus=1), cluster.functions=cluster.functions) register(param2) FUN <- function(i) system("hostname", intern=TRUE) xx <- bplapply(1:10, FUN) table(unlist(xx))
You should see results like this:
smallcluster-master smallcluster-node001 5 5
When you start a cluster using the above steps, R is automatically aware of the cluster, as shown in the following example:
With the default cluster configuration, this should return 2, which
make sense, since our cluster consists of two machines (
smallcluster-node001), each of type m1.small, which
have one core each.
BiocParallel, you can run a simple function
on your MPI cluster:
FUN <- function(i) system("hostname", intern=TRUE)
SnowParam instance with the number of nodes equal to
the size of the MPI universe minus 1 (let one node dispatch jobs to
workers), and register this instance as the default:
param3 <- SnowParam(mpi.universe.size() - 1, "MPI") register(param3)
Evaluate the work in parallel and process the results:
xx <- bplapply(1:10, FUN) table(unlist(xx))
Note: If you make changes to the running Bioconductor AMI, and then terminate the AMI, your changes will be lost. Use the steps described here to ensure that your changes are persistent.
If the AMI is missing some packages or features you think it should have, please let us know.
If you want to customize the AMI for your own purposes, it is simple. Just go ahead and customize your
running instance as you see fit. Typically this will involve installing R packages with
and software packages (at the operating system level) with the Ubuntu package manager
When your instance is customized to your liking, issue the following commands:
sudo clean_ami exit
This will remove unneeded files from your AMI, clear your shell history, and log you out of your instance. You may also want to change the password of the “ubuntu” user (because the default password is publicly known, in order to run RStudio Server) with the command:
Now use the AWS Console to Stop your instance (important note: do NOT “Terminate” your instance; use the Stop command (under Instance Actions) instead.)
Then choose “Create Image (EBS AMI)” under the Instance Actions menu. You will be prompted for a name for your AMI. After entering the name, your AMI will be created and given a unique AMI ID. You can then launch instances of this AMI using the steps above, being sure to substitute the ID of your own AMI. Your AMI will be private, accessible only to your AWS account, unless you decide to make it more widely accessible.
Now you should Terminate the Stopped instance of the Bioconductor AMI.
The Bioconductor AMI was created using Vagrant and Chef. The same scripts that were used to create these AMIs can also be used to provision virtual machines (Virtualbox or VMWare) or physical machines.
For more information, see the scripts’ github repository.
If you are using RStudio Server, you can upload and download files from the Files pane in RStudio server.
If you are connected via ssh, the scp command is the most efficient way to move data to and from your EC2 instances.
To copy a file from your computer to a running Bioconductor AMI instance:
scp -i bioconductor-bob-mylaptop.pem /path/to/myfile email@example.com:~
That will copy the file at “/path/to/myfile” on your local computer to ubuntu’s home directory on the remote instance. To copy a file from a running instance to your local computer, do something like this (still at your local computer):
scp -i bioconductor-bob-mylaptop.pem firstname.lastname@example.org:~/myfile /some/directory
That will copy the file ~/myfile from the running instance to /some/directory on your local machine.
Reminder: Files created on a running EC2 instance are not persisted unless you do some special steps. So if you are generating output files with Bioconductor, you must copy them to your local machine before terminating your instance, or your files will be lost. If you have a lot of data to move back and forth, you may want to look into Elastic Block Storage.
If you have questions about the Bioconductor AMI, please contact us through the Bioconductor support site.