ReUseData is an R/Bioconductor software tool to provide a systematic and versatile approach for standardized and reproducible data management. ReUseData facilitates transformation of shell or other ad hoc scripts for data preprocessing into workflow-based data recipes. Evaluation of data recipes generate curated data files in their generic formats (e.g., VCF, bed). Both recipes and data are cached using database infrastructure for easy data management and reuse. Prebuilt data recipes are available through ReUseData portal (“https://rcwl.org/dataRecipes/”) with full annotation and user instructions. Pregenerated data are available through ReUseData cloud bucket that is directly downloadable through “getCloudData()”.
This quick start shows the basic use of package functions in 2 major categories for managing:
Details for each section can be found in the companion vignettes for data recipes and reusable data.
BiocManager::install(c("ReUseData", "Rcwl"))library(ReUseData)All pre-built data recipes are included in the package and can be
easily updated (recipeUpdate), searched (recipeSearch) and loaded
(recipeLoad). Details about data recipes can be found in the
vignette ReUseData_recipe.html.
recipeUpdate(cachePath = "ReUseDataRecipe", force = TRUE)
#> NOTE: existing caches will be removed and regenerated!
#> Updating recipes...
#> STAR_index.R added
#> bowtie2_index.R added
#> echo_out.R added
#> ensembl_liftover.R added
#> gcp_broad_gatk_hg19.R added
#> gcp_broad_gatk_hg38.R added
#> gcp_gatk_mutect2_b37.R added
#> gcp_gatk_mutect2_hg38.R added
#> gencode_annotation.R added
#> gencode_genome_grch38.R added
#> gencode_transcripts.R added
#> hisat2_index.R added
#> reference_genome.R added
#> salmon_index.R added
#> ucsc_database.R added
#>
#> recipeHub with 15 records
#> cache path: /tmp/RtmpgEiA85/cache/ReUseDataRecipe
#> # recipeSearch() to query specific recipes using multipe keywords
#> # recipeUpdate() to update the local recipe cache
#>
#> name
#> BFC16 | STAR_index
#> BFC17 | bowtie2_index
#> BFC18 | echo_out
#> BFC19 | ensembl_liftover
#> BFC20 | gcp_broad_gatk_hg19
#> ... ...
#> BFC26 | gencode_transcripts
#> BFC27 | hisat2_index
#> BFC28 | reference_genome
#> BFC29 | salmon_index
#> BFC30 | ucsc_database
recipeSearch("echo")
#> recipeHub with 1 records
#> cache path: /tmp/RtmpgEiA85/cache/ReUseDataRecipe
#> # recipeSearch() to query specific recipes using multipe keywords
#> # recipeUpdate() to update the local recipe cache
#>
#> name
#> BFC18 | echo_out
echo_out <- recipeLoad("echo_out")
#> Note: you need to assign a name for the recipe: rcpName <- recipeLoad('xx')
#> Data recipe loaded!
#> Use inputs() to check required input parameters before evaluation.
#> Check here: https://rcwl.org/dataRecipes/echo_out.html
#> for user instructions (e.g., eligible input values, data source, etc.)We can install cwltool first to make sure a cwl-runner is available.
invisible(Rcwl::install_cwltool())A data recipe can be evaluated by assigning values to the recipe
parameters. getData runs the recipe as a CWL scripts internally, and
generates the data of interest with annotation files for future reuse.
Rcwl::inputs(echo_out)
#> inputs:
#> input (input) (string):
#> outfile (outfile) (string):
echo_out$input <- "Hello World!"
echo_out$outfile <- "outfile"
outdir <- file.path(tempdir(), "SharedData")
res <- getData(echo_out,
outdir = outdir,
notes = c("echo", "hello", "world", "txt"))
#> }[1;30mINFO[0m Final process status is success
res$out
#> [1] "/tmp/RtmpgEiA85/SharedData/outfile.txt"
readLines(res$out)
#> [1] "Print the input: Hello World!"One can create a data recipe from scratch or by converting an existing
shell script for data processing, by specifying input parameters,
output globbing patterns using recipeMake function.
script <- system.file("extdata", "echo_out.sh", package = "ReUseData")
rcp <- recipeMake(shscript = script,
paramID = c("input", "outfile"),
paramType = c("string", "string"),
outputID = "echoout",
outputGlob = "*.txt")
Rcwl::inputs(rcp)
#> inputs:
#> input (string):
#> outfile (string):
Rcwl::outputs(rcp)
#> outputs:
#> echoout:
#> type: File[]
#> outputBinding:
#> glob: '*.txt'The data that are generated from evaluating data recipes are
automatically annotated and tracked with user-specified keywords and
time/date tags. It uses a similar cache system as for recipes for
users to easily update (dataUpdate), search (dataSearch) and use
(toList).
Pre-generated data files from existing data recipes are saved in
Google Cloud Bucket, that are ready to be queried
(dataSearch(cloud=TRUE)) and downloaded (getCloudData) to local
cache system with annotations.
ReUseDatadh <- dataUpdate(dir = outdir)
dataSearch(c("echo", "hello"))
dataNames(dh)
dataParams(dh)
dataNotes(dh)toList(dh, format="json", file = file.path(outdir, "data.json"))dh <- dataUpdate(dir = outdir, cloud = TRUE)
getCloudData(dh[2], outdir = outdir)sessionInfo()
#> R version 4.3.2 Patched (2023-11-13 r85521)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 22.04.3 LTS
#>
#> Matrix products: default
#> BLAS: /home/biocbuild/bbs-3.18-bioc/R/lib/libRblas.so
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_GB LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: America/New_York
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] ReUseData_1.2.2 Rcwl_1.18.0 S4Vectors_0.40.2
#> [4] BiocGenerics_0.48.1 yaml_2.3.7 BiocStyle_2.30.0
#>
#> loaded via a namespace (and not attached):
#> [1] tidyselect_1.2.0 dplyr_1.1.4 blob_1.2.4
#> [4] filelock_1.0.2 R.utils_2.12.3 fastmap_1.1.1
#> [7] BiocFileCache_2.10.1 promises_1.2.1 digest_0.6.33
#> [10] base64url_1.4 mime_0.12 lifecycle_1.0.4
#> [13] ellipsis_0.3.2 RSQLite_2.3.4 magrittr_2.0.3
#> [16] compiler_4.3.2 rlang_1.1.2 sass_0.4.8
#> [19] progress_1.2.3 tools_4.3.2 utf8_1.2.4
#> [22] data.table_1.14.10 knitr_1.45 prettyunits_1.2.0
#> [25] brew_1.0-8 htmlwidgets_1.6.4 bit_4.0.5
#> [28] curl_5.2.0 reticulate_1.34.0 RColorBrewer_1.1-3
#> [31] batchtools_0.9.17 BiocParallel_1.36.0 purrr_1.0.2
#> [34] withr_2.5.2 R.oo_1.25.0 grid_4.3.2
#> [37] fansi_1.0.6 git2r_0.33.0 xtable_1.8-4
#> [40] debugme_1.1.0 cli_3.6.1 rmarkdown_2.25
#> [43] DiagrammeR_1.0.10 crayon_1.5.2 generics_0.1.3
#> [46] httr_1.4.7 visNetwork_2.1.2 DBI_1.1.3
#> [49] cachem_1.0.8 parallel_4.3.2 BiocManager_1.30.22
#> [52] basilisk_1.14.1 vctrs_0.6.5 Matrix_1.6-4
#> [55] jsonlite_1.8.8 dir.expiry_1.10.0 bookdown_0.37
#> [58] hms_1.1.3 bit64_4.0.5 jquerylib_0.1.4
#> [61] RcwlPipelines_1.18.0 glue_1.6.2 codetools_0.2-19
#> [64] stringi_1.8.2 later_1.3.2 tibble_3.2.1
#> [67] pillar_1.9.0 basilisk.utils_1.14.1 rappdirs_0.3.3
#> [70] htmltools_0.5.7 R6_2.5.1 dbplyr_2.4.0
#> [73] evaluate_0.23 shiny_1.8.0 lattice_0.22-5
#> [76] R.methodsS3_1.8.2 png_0.1-8 backports_1.4.1
#> [79] memoise_2.0.1 httpuv_1.6.13 bslib_0.6.1
#> [82] Rcpp_1.0.11 checkmate_2.3.1 xfun_0.41
#> [85] pkgconfig_2.0.3