Note: the most recent version of this tutorial can be found here and a short overview slide show here.

Note: if you use systemPipeR in published research, please cite: Backman, T.W.H and Girke, T. (2016). systemPipeR: NGS Workflow and Report Generation Environment. BMC Bioinformatics, 17: 388. 10.1186/s12859-016-1241-0.

1 Introduction

systemPipeR provides utilities for building and running automated end-to-end analysis workflows for a wide range of research applications, including next generation sequence (NGS) experiments, such as RNA-Seq, ChIP-Seq, VAR-Seq and Ribo-Seq (H Backman and Girke 2016)]. Important features include a uniform workflow interface across different data analysis applications, automated report generation, and support for running both R and command-line software, such as NGS aligners or peak/variant callers, on local computers or compute clusters (Figure 1). The latter supports interactive job submissions and batch submissions to queuing systems of clusters. For instance, systemPipeR can be used with most command-line aligners such as BWA (Li 2013; Li and Durbin 2009), HISAT2 (Kim, Langmead, and Salzberg 2015), TopHat2 (Kim et al. 2013) and Bowtie2 (Langmead and Salzberg 2012), as well as the R-based NGS aligners Rsubread (Liao, Smyth, and Shi 2013) and gsnap (gmapR) (Wu and Nacu 2010). Efficient handling of complex sample sets (e.g. FASTQ/BAM files) and experimental designs is facilitated by a well-defined sample annotation infrastructure which improves reproducibility and user-friendliness of many typical analysis workflows in the NGS area (Lawrence et al. 2013).

The main motivation and advantages of using systemPipeR for complex data analysis tasks are:

  1. Facilitates design of complex NGS workflows involving multiple R/Bioconductor packages
  2. Common workflow interface for different NGS applications
  3. Makes NGS analysis with Bioconductor utilities more accessible to new users
  4. Simplifies usage of command-line software from within R
  5. Reduces complexity of using compute clusters for R and command-line software
  6. Accelerates runtime of workflows via parallelzation on computer systems with mutiple CPU cores and/or multiple compute nodes
  7. Automates generation of analysis reports to improve reproducibility

Figure 1: Relevant features in systemPipeR. Workflow design concepts are illustrated under (A & B). Examples of systemPipeR’s visualization functionalities are given under (C).

A central concept for designing workflows within the sytemPipeR environment is the use of workflow management containers. In previous versions, sytemPipeR used a custom command-line interface called SYSargs (see Figure 2) and for this purpose will continue to be supported for some time. With the latest Bioconductor Release 3.9, we are adopting for this functionality the widely used community standard Common Workflow Language (CWL) for describing analysis workflows in a generic and reproducible manner, introducing SYSargs2 workflow control class (see Figure 3). Using this community standard in sytemPipeR has many advantages. For instance, the integration of CWL allows running sytemPipeR workflows from a single specification instance either entirely from within R, from various command-line wrappers (e.g., cwl-runner) or from other languages (, e.g., Bash or Python). sytemPipeR includes support for both command-line and R/Bioconductor software as well as resources for containerization, parallel evaluations on computer clusters along with the automated generation of interactive analysis reports.

An important feature of sytemPipeR's CWL interface is that it provides two options to run command-line tools and workflows based on CWL. First, one can run CWL in its native way via an R-based wrapper utility for cwl-runner or cwl-tools (CWL-based approach). Second, one can run workflows using CWL’s command-line and workflow instructions from within R (R-based approach). In the latter case the same CWL workflow definition files (e.g. *.cwl and *.yml) are used but rendered and exectuted entirely with R functions defined by sytemPipeR, and thus use CWL mainly as a command-line and workflow definition format rather than a software to run workflows. In this regard sytemPipeR also provides several convenience functions that are useful for designing and debugging workflows, such as a command-line rendering function to retrieve the exact command-line strings for each data set and processing step prior to running a command-line.

This tutorial introduces the design of a new CWL S4 class in systemPipeR, as well as the custom command-line interface, combined with the overview of all the common analysis steps of NGS experiments.

1.1 Workflow design structure using SYSargs

Instances of this S4 object class are constructed by the systemArgs function from two simple tabular files: a targets file and a param file. The latter is optional for workflow steps lacking command-line software. Typically, a SYSargs instance stores all sample-level inputs as well as the paths to the corresponding outputs generated by command-line- or R-based software generating sample-level output files, such as read preprocessors (trimmed/filtered FASTQ files), aligners (SAM/BAM files), variant callers (VCF/BCF files) or peak callers (BED/WIG files). Each sample level input/outfile operation uses its own SYSargs instance. The outpaths of SYSargs usually define the sample inputs for the next SYSargs instance