Chapter 4 Annotation diagnostics

4.1 Overview

In addition to the labels, SingleR() returns a number of helpful diagnostics about the annotation process that can be used to determine whether the assignments are appropriate. Unambiguous assignments corroborated by expression of canonical markers add confidence to the results; conversely, low-confidence assignments can be pruned out to avoid adding noise to downstream analyses. This chapter will demonstrate some of these common sanity checks on the pancreas datasets from Chapter 3 (Muraro et al. 2016; Grun et al. 2016).

4.2 Based on the scores within cells

The most obvious diagnostic reported by SingleR() is the nested matrix of per-cell scores in the scores field. This contains the correlation-based scores prior to any fine-tuning for each cell (row) and reference label (column). Ideally, we would see unambiguous assignments where, for any given cell, one label’s score is clearly larger than the others.

##       acinar  alpha   beta  delta   duct endothelial epsilon mesenchymal
##  [1,] 0.6797 0.1463 0.1554 0.1029 0.4971      0.2030 0.04434      0.1848
##  [2,] 0.6832 0.1849 0.1611 0.1250 0.5133      0.2231 0.08909      0.2140
##  [3,] 0.6845 0.2032 0.2141 0.1875 0.5060      0.2286 0.08960      0.1961
##  [4,] 0.6472 0.2050 0.2115 0.1879 0.7300      0.2285 0.12107      0.2708
##  [5,] 0.6079 0.2285 0.2419 0.1938 0.6616      0.2659 0.16007      0.3228
##  [6,] 0.5816 0.2789 0.2611 0.2661 0.6827      0.2857 0.21781      0.3151
##  [7,] 0.5704 0.3268 0.2990 0.2618 0.6293      0.2947 0.20834      0.3263
##  [8,] 0.5692 0.2455 0.2152 0.2033 0.6495      0.2652 0.18469      0.2943
##  [9,] 0.6421 0.2101 0.1929 0.1894 0.7201      0.2795 0.14615      0.3142
## [10,] 0.7153 0.2109 0.1747 0.1606 0.5896      0.1687 0.15639      0.1778
##            pp
##  [1,] 0.08954
##  [2,] 0.11207
##  [3,] 0.15207
##  [4,] 0.14351
##  [5,] 0.18475
##  [6,] 0.21898
##  [7,] 0.25871
##  [8,] 0.19160
##  [9,] 0.17281
## [10,] 0.16314

To check whether this is indeed the case, we use the plotScoreHeatmap() function to visualize the score matrix (Figure 4.1). Here, the key is to examine the spread of scores within each cell, i.e., down the columns of the heatmap. Similar scores for a group of labels indicates that the assignment is uncertain for those columns, though this may be acceptable if the uncertainty is distributed across closely related cell types. (Note that the assigned label for a cell may not be the visually top-scoring label if fine-tuning is applied, as the only the pre-tuned scores are directly comparable across all labels.)

Heatmap of normalized scores for the Grun dataset. Each cell is a column while each row is a label in the reference Muraro dataset. The final label (after fine-tuning) for each cell is shown in the top color bar.

Figure 4.1: Heatmap of normalized scores for the Grun dataset. Each cell is a column while each row is a label in the reference Muraro dataset. The final label (after fine-tuning) for each cell is shown in the top color bar.

We can also display other metadata information for each cell by setting clusters= or annotation_col=. This is occasionally useful for examining potential batch effects, differences in cell type composition between conditions, relationship to clusters from an unsupervised analysis and so on,. For example, Figure 4.2 displays the donor of origin for each cell; we can see that each cell type has contributions from multiple donors, which is reassuring as it indicates that our assignments are not (purely) driven by donor effects.

Heatmap of normalized scores for the Grun dataset, including the donor of origin for each cell.

Figure 4.2: Heatmap of normalized scores for the Grun dataset, including the donor of origin for each cell.

The scores matrix has several caveats associated with its interpretation. Only the pre-tuned scores are stored in this matrix, as scores after fine-tuning are not comparable across all labels. This means that the label with the highest score for a cell may not be the cell’s final label if fine-tuning is applied. Moreover, the magnitude of differences in the scores has no clear interpretation; indeed, plotScoreHeatmap() dispenses with any faithful representation of the scores and instead adjusts the values to highlight any differences between labels within each cell.

4.3 Based on the deltas across cells

We identify poor-quality or ambiguous assignments based on the per-cell “delta”, i.e., the difference between the score for the assigned label and the median across all labels for each cell. Our assumption is that most of the labels in the reference are not relevant to any given cell. Thus, the median across all labels can be used as a measure of the baseline correlation, while the gap from the assigned label to this baseline can be used as a measure of the assignment confidence.

Low deltas indicate that the assignment is uncertain, possibly because the cell’s true label does not exist in the reference. An obvious next step is to apply a threshold on the delta to filter out these low-confidence assignments. We use the delta rather than the assignment score as the latter is more sensitive to technical effects. For example, changes in library size affect the technical noise and can increase/decrease all scores for a given cell, while the delta is somewhat more robust as it focuses on the differences between scores within each cell.

SingleR() will set a threshold on the delta for each label using an outlier-based strategy. Specifically, we identify cells with deltas that are small outliers relative to the deltas of other cells with the same label. This assumes that, for any given label, most cells assigned to that label are correct. We focus on outliers to avoid difficulties with setting a fixed threshold, especially given that the magnitudes of the deltas are about as uninterpretable as the scores themselves. Pruned labels are reported in the pruned.labels field where low-quality assignments are replaced with NA.

##              Removed
## Label         FALSE TRUE
##   acinar        251   26
##   alpha         198    5
##   beta          180    1
##   delta          49    1
##   duct          301    5
##   endothelial     4    1
##   epsilon         1    0
##   mesenchymal    22    0
##   pp             17    2

However, the default pruning parameters may not be appropriate for every dataset. For example, if one label is consistently misassigned, the assumption that most cells are correctly assigned will not be appropriate. In such cases, we can revert to a fixed threshold by manually calling the underlying pruneScores() function with min.diff.med=. The example below discards cells with deltas below an arbitrary threshold of 0.2, where higher thresholds correspond to greater assignment certainty.

##              Removed
## Label         FALSE TRUE
##   acinar        250   27
##   alpha         155   48
##   beta          148   33
##   delta          33   17
##   duct          301    5
##   endothelial     4    1
##   epsilon         0    1
##   mesenchymal    22    0
##   pp              4   15

This entire process can be visualized using the plotScoreDistribution() function, which displays the per-label distribution of the deltas across cells (Figure 4.3). We can use this plot to check that outlier detection in pruneScores() behaved sensibly. Labels with especially low deltas may warrant some additional caution in their interpretation.

Distribution of deltas for the Grun dataset. Each facet represents a label in the Muraro dataset, and each point represents a cell assigned to that label (colored by whether it was pruned).

Figure 4.3: Distribution of deltas for the Grun dataset. Each facet represents a label in the Muraro dataset, and each point represents a cell assigned to that label (colored by whether it was pruned).

If fine-tuning was performed, we can apply an even more stringent filter based on the difference between the highest and second-highest scores after fine-tuning. Cells will only pass the filter if they are assigned to a label that is clearly distinguishable from any other label. In practice, this approach tends to be too conservative as assignments involving closely related labels are heavily penalized.

##              Removed
## Label         FALSE TRUE
##   acinar        235   42
##   alpha         166   37
##   beta          117   64
##   delta          25   25
##   duct          157  149
##   endothelial     4    1
##   epsilon         0    1
##   mesenchymal    22    0
##   pp              9   10

4.4 Based on marker gene expression

Another simple yet effective diagnostic is to examine the expression of the marker genes for each label in the test dataset. The marker genes used for each label are reported in the metadata() of the SingleR() output, so we can simply retrieve them to visualize their (usually log-transformed) expression values across the test dataset. In Figure 4.4, we use the plotHeatmap() function from scater to examine the expression of markers used to identify beta cells.

Heatmap of log-expression values in the Grun dataset for all marker genes upregulated in beta cells in the Muraro reference dataset. Assigned labels for each cell are shown at the top of the plot.

Figure 4.4: Heatmap of log-expression values in the Grun dataset for all marker genes upregulated in beta cells in the Muraro reference dataset. Assigned labels for each cell are shown at the top of the plot.

If a cell in the test dataset is confidently assigned to a particular label, we would expect it to have strong expression of that label’s markers. We would also hope that those label’s markers are biologically meaningful; in this case, we do observe strong upregulation of insulin (INS) in the beta cells, which is reassuring and gives greater confidence to the correctness of the assignment. If the identified markers are not meaningful or not consistently upregulated, some skepticism towards the quality of the assignments is warranted.

In practice, the heatmap may be overwhelmingly large if there too many reference-derived markers. To resolve this, we can prune the set of markers to focus on the most interesting genes based on their test expression profiles. Figure 4.5 is limited to the top genes with the strongest evidence for upregulation in our test dataset using the assigned labels; such genes are effectively markers for beta cells in both the reference and test datasets. As a diagnostic plot, this is much more amenable to quick inspection to check that the expected genes are present.

Heatmap of log-expression values in the Grun dataset for all marker genes upregulated in beta cells in the Muraro reference dataset, pruned to those that are also upregulated in the assigned cells in the Grun dataset. Assigned labels for each cell are shown at the top of the plot.

Figure 4.5: Heatmap of log-expression values in the Grun dataset for all marker genes upregulated in beta cells in the Muraro reference dataset, pruned to those that are also upregulated in the assigned cells in the Grun dataset. Assigned labels for each cell are shown at the top of the plot.

It is straightforward to repeat this process for all labels by wrapping this code in a loop, as shown below in Figure 4.6. Note that plotHeatmap() is not the only function that can be used for this visualization; we could also use plotDots() to create a Seurat-style dot plot, or we could use other heatmap plotting functions such as dittoHeatmap() from dittoSeq.

Heatmaps of log-expression values in the Grun dataset for all marker genes upregulated in each label in the Muraro reference dataset. Assigned labels for each cell are shown at the top of each plot.

Figure 4.6: Heatmaps of log-expression values in the Grun dataset for all marker genes upregulated in each label in the Muraro reference dataset. Assigned labels for each cell are shown at the top of each plot.

In general, the heatmap provides a more interpretable diagnostic visualization than the plots of scores and deltas. However, it does require more effort to inspect and may not be feasible for large numbers of labels. It is also difficult to use a heatmap to determine the correctness of assignment for closely related labels.

4.5 Comparing to unsupervised clustering

It can also be instructive to compare the assigned labels to the groupings generated from unsupervised clustering algorithms. The assumption is that the differences between reference labels are also the dominant factor of variation in the test dataset; this implies that we should expect strong agreement between the clusters and the assigned labels. To demonstrate, we’ll use the sceG from Chapter 8 where clusters have generated using a graph-based method (Xu and Su 2015).

We compare these clusters to the labels generated by SingleR. Any similarity can be quantified with the adjusted rand index (ARI) with pairwiseRand() from the bluster package. Large ARIs indicate that the two partitionings are in agreement, though an acceptable definition of “large” is difficult to gauge; experience suggests that a reasonable level of consistency is achieved at ARIs above 0.5.

## [1] 0.3922

In practice, it is more informative to examine the distribution of cells across each cluster/label combination. Figure ?? shows that most clusters are nested within labels, a difference in resolution that is likely responsible for reducing the ARI. Clusters containing multiple labels are particularly interesting for diagnostic purposes, as this suggests that the differences between labels are not strong enough to drive formation of distinct clusters in the test.

Heatmap of the log-transformed number of cells in each combination of label (column) and cluster (row) in the Grun dataset.

Figure 4.7: Heatmap of the log-transformed number of cells in each combination of label (column) and cluster (row) in the Grun dataset.

The underlying assumption is somewhat reasonable in most scenarios where the labels relate to cell type identity. However, disagreements between the clusters and labels should not be cause for much concern. The whole point of unsupervised clustering is to identify novel variation that, by definition, is not in the reference. It is entirely possible for the clustering and labels to be different without compromising the validity or utility of either; the former captures new heterogeneity while the latter facilitates interpretation in the context of existing knowledge.

Session information

R version 4.0.4 (2021-02-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.2 LTS

Matrix products: default
BLAS:   /home/biocbuild/bbs-3.12-books/R/lib/libRblas.so
LAPACK: /home/biocbuild/bbs-3.12-books/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C              
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] parallel  stats4    stats     graphics  grDevices utils     datasets 
[8] methods   base     

other attached packages:
 [1] bluster_1.0.0               scran_1.18.5               
 [3] scater_1.18.6               ggplot2_3.3.3              
 [5] SingleCellExperiment_1.12.0 SingleR_1.4.1              
 [7] SummarizedExperiment_1.20.0 Biobase_2.50.0             
 [9] GenomicRanges_1.42.0        GenomeInfoDb_1.26.4        
[11] IRanges_2.24.1              MatrixGenerics_1.2.1       
[13] matrixStats_0.58.0          S4Vectors_0.28.1           
[15] BiocGenerics_0.36.0         BiocStyle_2.18.1           
[17] rebook_1.0.0               

loaded via a namespace (and not attached):
 [1] bitops_1.0-6              RColorBrewer_1.1-2       
 [3] tools_4.0.4               bslib_0.2.4              
 [5] utf8_1.2.1                R6_2.5.0                 
 [7] irlba_2.3.3               vipor_0.4.5              
 [9] DBI_1.1.1                 colorspace_2.0-0         
[11] withr_2.4.1               tidyselect_1.1.0         
[13] gridExtra_2.3             processx_3.4.5           
[15] compiler_4.0.4            graph_1.68.0             
[17] BiocNeighbors_1.8.2       DelayedArray_0.16.2      
[19] labeling_0.4.2            bookdown_0.21            
[21] sass_0.3.1                scales_1.1.1             
[23] callr_3.5.1               stringr_1.4.0            
[25] digest_0.6.27             rmarkdown_2.7            
[27] XVector_0.30.0            pkgconfig_2.0.3          
[29] htmltools_0.5.1.1         sparseMatrixStats_1.2.1  
[31] limma_3.46.0              highr_0.8                
[33] rlang_0.4.10              DelayedMatrixStats_1.12.3
[35] jquerylib_0.1.3           generics_0.1.0           
[37] farver_2.1.0              jsonlite_1.7.2           
[39] BiocParallel_1.24.1       dplyr_1.0.5              
[41] RCurl_1.98-1.3            magrittr_2.0.1           
[43] BiocSingular_1.6.0        GenomeInfoDbData_1.2.4   
[45] scuttle_1.0.4             Matrix_1.3-2             
[47] ggbeeswarm_0.6.0          Rcpp_1.0.6               
[49] munsell_0.5.0             fansi_0.4.2              
[51] viridis_0.5.1             lifecycle_1.0.0          
[53] edgeR_3.32.1              stringi_1.5.3            
[55] yaml_2.2.1                zlibbioc_1.36.0          
[57] grid_4.0.4                dqrng_0.2.1              
[59] crayon_1.4.1              lattice_0.20-41          
[61] beachmat_2.6.4            locfit_1.5-9.4           
[63] CodeDepends_0.6.5         knitr_1.31               
[65] ps_1.6.0                  pillar_1.5.1             
[67] igraph_1.2.6              codetools_0.2-18         
[69] XML_3.99-0.6              glue_1.4.2               
[71] evaluate_0.14             BiocManager_1.30.10      
[73] vctrs_0.3.6               gtable_0.3.0             
[75] purrr_0.3.4               assertthat_0.2.1         
[77] xfun_0.22                 rsvd_1.0.3               
[79] viridisLite_0.3.0         tibble_3.1.0             
[81] pheatmap_1.0.12           beeswarm_0.3.1           
[83] statmod_1.4.35            ellipsis_0.3.1

Bibliography

Grun, D., M. J. Muraro, J. C. Boisset, K. Wiebrands, A. Lyubimova, G. Dharmadhikari, M. van den Born, et al. 2016. “De Novo Prediction of Stem Cell Identity using Single-Cell Transcriptome Data.” Cell Stem Cell 19 (2): 266–77.

Muraro, M. J., G. Dharmadhikari, D. Grun, N. Groen, T. Dielen, E. Jansen, L. van Gurp, et al. 2016. “A Single-Cell Transcriptome Atlas of the Human Pancreas.” Cell Syst 3 (4): 385–94.

Xu, C., and Z. Su. 2015. “Identification of cell types from single-cell transcriptomes using a novel clustering method.” Bioinformatics 31 (12): 1974–80.