1.1 Highly multiplexed imaging

Highly multiplexed imaging allows simultaneous spatially and single-cell resolved detection of dozens of biological molecules (e.g. proteins) in their native tissue context. As a result, these technologies allow an in-depth analysis of complex systems and diseases such as the tumor microenvironment (Jackson et al. 2020) and type 1 diabetes progression (Damond et al. 2019).

Imaging-based spatial proteomics methods (Moffitt, Lundberg, and Heyn 2022) can be broadly divided into fluorescent cyclic approaches such as tissue-based cyclic immunofluorescence (t-CyCIF) (Lin et al. 2018) and one-step mass-tag based approaches such as multiplexed ion beam imaging (MIBI) (Angelo et al. 2014) and IMC (Giesen et al. 2014).

Of note, the instructions below will focus on the visualization and exploration of IMC data as an example. However, data from other technologies such as t-CyCIF or MIBI, which produce pixel-level intensities and (optionally) segmentation masks, can be interactively visualized with cytoviewer as long as they have the appropriate input format (see Section Data input format).

1.2 Highly multiplexed image analysis

To fully leverage the information contained in IMC and multiplexed imaging data in general, computational tools are of key importance.

The main analysis steps, irrespective of the biological question, include 1) Visual inspection of images for quality control, 2) Image pre-processing and segmentation and 3) Single-cell and spatial analysis (Windhager, Bodenmiller, and Eling 2021).

A comprehensive end-to-end workflow for multiplexed image processing and analysis with detailed information for every analysis step can be found here.

Importantly, the cytoviewer package can support, simplify and improve any of these analysis steps with its easy-to-use interactive visualization interface in R.

Below we will showcase an example workflow that highlights the different functionality and potential application fields of cytoviewer.

1.3 Application overview

The cytoviewer interface is broadly divided into image-level (Composite and Channels) and cell-level visualization (Masks). It allows users to overlay individual images with segmentation masks, integrates well with SingleCellExperiment and SpatialExperiment objects for metadata visualization and supports image downloads (Figure 2B).

Figure 1: cytoviewer interface and functionality. (A) The supported functionality (right) of cytoviewer depends on the data inputs (left). To match information between the objects, cell (cell_id) and image (img_id) identifiers can be provided. SCE/SPE = SingleCellExperiment/SpatialExperiment. (B) The graphical user interface of cytoviewer is divided into a body, header, and sidebar. The body of cytoviewer includes the image viewer, which has three tabs: Composite (Image-level), Channels (Image-level), and Mask (Cell-level). Zooming is supported for Composite and Mask tabs. The package version, R session information, help page, and a drop-down menu for image downloads are located in the header. The sidebar menu has controls for sample selection, image visualization, mask visualization, and general settings. Scale bar: 150 µm (C) cytoviewer supports different viewing modes. Top: The “channels” tab of image-level visualization displays individual channels. Shown are Ecad (magenta), CD8a (cyan), and CD68 (yellow) marking tumor cells, CD8+ T cells, and myeloid cells, respectively. Center: The “composite” tab of image-level visualization visualizes image composites combining multiple channels. These composite images can be overlayed with cell outlines, which can be colored by cell-specific metadata. Shown here are cell outlines colored by cell area (continous value) and cell type (categorical value; tumor cells in white). Channel color settings are as follows for all markers: Contrast: 2,5; Brightness: 1; Gamma: 1.2. Bottom: The “mask” tab can be used to visualize segmentation masks that can be colored by cell-specific metadata. Shown here are segmentation masks colored by cell area (continuous) and cell type (categorical; tumor cells in magenta). Scale bars: 150 µm. (D) “Image appearance” controls can be used to add legends or titles and to change the scale bar length for image-level (top) and cell level (bottom) visualization. The cell-level mask plot depicts tumor (magenta), myeloid (yellow), and CD8+ T cells (cyan). Scale bars: 100 µm. Adapted from (Meyer, Eling, and Bodenmiller 2023)

2.1 Installation

The cytoviewer package can be installed from Bioconductor via:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("cytoviewer")

The development version of cytoviewer can be installed from Github via:

if (!requireNamespace("remotes", quietly = TRUE))
    install.packages("remotes")

remotes::install_github("BodenmillerGroup/cytoviewer")

To load the package in your R session, type the following:

library(cytoviewer)

2.2 Example dataset

For visualization purposes, we will use a toy dataset provided by the cytomapper package.

The dataset contains 3 images of $100\mu{m}$ x $100\mu{m}$ dimensions with 362 segmented cells and pixel-intensities for 5 proteins: H3, CD99, PIN, CD8a, and CDH. It is a small subset from a Type 1 Diabetes dataset (Damond et al. 2019).

Pixel-level intensities for all 5 markers (5 channels) are stored in the pancreasImages object.

The corresponding segmentation masks are stored in the pancreasMasks object.

All cell-specific metadata are stored in the colData slot of the corresponding SingleCellExperiment object: pancreasSCE.

For more detailed information on the dataset, please refer to the respective documentation (e.g. via ?pancreasImages or the vignette of the cytomapper package).

We also provide example code to directly read in images and masks (e.g. in .tiff format) into a CytoImageList object and create a SingleCellExperiment object from them in the Read in data section.

# Load example datasets 
library(cytomapper)
data("pancreasImages")
data("pancreasMasks")
data("pancreasSCE")

pancreasImages

## CytoImageList containing 3 image(s)
## names(3): E34_imc G01_imc J02_imc 
## Each image contains 5 channel(s)
## channelNames(5): H3 CD99 PIN CD8a CDH

pancreasMasks 

## CytoImageList containing 3 image(s)
## names(3): E34_mask G01_mask J02_mask 
## Each image contains 1 channel

pancreasSCE

## class: SingleCellExperiment 
## dim: 5 362 
## metadata(0):
## assays(2): counts exprs
## rownames(5): H3 CD99 PIN CD8a CDH
## rowData names(4): MetalTag Target clean_Target frame
## colnames(362): E34_824 E34_835 ... J02_4190 J02_4209
## colData names(9): ImageName Pos_X ... MaskName Pattern
## reducedDimNames(0):
## mainExpName: NULL
## altExpNames(0):

2.3 Function call

Here as an example, we call cytoviewer with image, mask and object data to leverage all provided functionality.

This setting allows image-level visualization (Composite and Channels), cell-level visualization, overlaying images with segmentation masks as well as metadata visualization.

For further details, please refer to the ?cytoviewer manual or the Help page within the shiny application.

# Use cytoviewer with images, masks and object
app <- cytoviewer(image = pancreasImages, 
                  mask = pancreasMasks, 
                  object = pancreasSCE, 
                  img_id = "ImageNb", 
                  cell_id = "CellNb")

if (interactive()) {
  
  shiny::runApp(app, launch.browser = TRUE)

  }

2.4 Interface

The cytoviewer interface is divided into a Header, Sidebar and Body section (see Figure below).

The Header includes package version information, access to session information and the help page as well as a dropdown-menu for image downloads.

The Body features a Tabset-Panel layout allowing the user to switch between three image modes: Image-level (Composite and Channels) and Cell-level (Mask). Furthermore, the Composite and Mask tabs have zoom controls.

The Sidebar panel is subdivided into four sections: Sample selection, Image-level, Cell-level and General controls.

2.5 Image-level visualization

Image visualization control is split into basic and advanced controls.

Basic controls supports the selection of up to six markers/channels for image display. Each marker has color control settings that allow the user to set contrast, brightness, gamma and select a channel color.

Figure 2: Image-level visualization - Basic controls. The graphical user interface of cytoviewer for image-level-composite with basic controls. For image-level visualization, Ecad (magenta), CD8a (cyan) and CD68 (yellow) marking tumor cells, CD8+ T cells and myeloid cells, respectively, are shown and channel color settings are as follows for all markers: Contrast: 2,5; Brightness: 1; Gamma: 1.2. Note that the Composite tab is zoomable. Scale bars: 150 µm. Adapted from (Meyer, Eling, and Bodenmiller 2023)

In the advanced controls part, the user can choose to overlay the displayed images with provided segmentation masks. Outline color and mask thickness can be adjusted by the user. Moreover, the masks can be outlined by cell-specific metadata provided in colData slot of the object.

Of note, for categorical and continuous metadata entries the user can choose between discrete colors and continuous color palettes (viridis, inferno, plasma), respectively.

Figure 3: Image-level visualization - Advanced controls. The graphical user interface of cytoviewer for image-level-composite with advanced controls. For image-level visualization, Ecad (magenta), CD8a (cyan) and CD68 (yellow) marking tumor cells, CD8+ T cells and myeloid cells, respectively, are shown and channel color settings are as follows for all markers: Contrast: 2,5; Brightness: 1; Gamma: 1.2. Note that the Composite tab is zoomable. Scale bars: 150 µm. Adapted from (Meyer, Eling, and Bodenmiller 2023)

2.6 Cell-level visualization

Cell visualization has basic controls.

Here, the user can choose to display the provided segmentation masks. If an object is provided, the masks can be colored by cell-specific metadata.

Please note again that for categorical and continuous metadata entries the user can choose between discrete colors and continuous color palettes (viridis, inferno, plasma), respectively.

Figure 4: Cell-level visualization - Basic controls. The graphical user interface of cytoviewer for cell-level-mask with basic controls. For cell-level visualization, tumor cells (magenta) are highlighted. Note that the Mask tab is zoomable. Adapted from (Meyer, Eling, and Bodenmiller 2023)

2.7 General controls

General controls is subdivided into an Image appearance and Image filters part.

In the Image appearance section, the user can adjust the scale bar length and include legend/image titles, while the Image filters section allows to control pixel-wise interpolation (default) and apply a Gaussian filter.

2.8 Image download

The cytoviewer package supports fast and uncomplicated image downloads.

Download controls are part of the Header (see Section Interface).

The user can specify a file name, select the image of interest (Composite, Channels, Mask) and the file format (pdf, png). Upon clicking the download button, a pop-window should appear where the user can specify the download location.

3.1 Read in data

To conveniently read in images and segmentation masks into a CytoImageList object and then visualize these using cytoviewer, the cytomapper package provides the loadImages function.

The loadImages function returns a CytoImageList object containing the multi-channel images or segmentation masks. Refer to the ?loadImages function to see the full functionality.

As an example, we will read in multi-channel images and segmentation masks provided by the cytomapper package.

To correctly scale pixel values of the segmentation masks when reading them in, we will need to set as.is = TRUE. Users needs to take care that pixel values are scaled correctly in more complex cases.

library(cytomapper)

# Data directory that stores images and masks in tiff format
data_path <- system.file("extdata", package = "cytomapper")

# Read in images
cur_images <- loadImages(data_path, pattern = "_imc.tiff")
cur_images

## CytoImageList containing 3 image(s)
## names(3): E34_imc G01_imc J02_imc 
## Each image contains 5 channel(s)

# Read in masks
cur_masks <- loadImages(data_path, pattern = "_mask.tiff", as.is = TRUE)
cur_masks

## CytoImageList containing 3 image(s)
## names(3): E34_mask G01_mask J02_mask 
## Each image contains 1 channel

names(cur_images)

## [1] "E34_imc" "G01_imc" "J02_imc"

names(cur_masks)

## [1] "E34_mask" "G01_mask" "J02_mask"

mcols(cur_masks)$ImageNb <- mcols(cur_images)$ImageNb <- c("E34", "G01", "J02")

channelNames(cur_images) <- c("H3", "CD99", "PIN", "CD8a", "CDH")

cur_sce <- measureObjects(image = cur_images, 
                          mask = cur_masks, 
                          img_id = "ImageNb")
cur_sce

## class: SingleCellExperiment 
## dim: 5 362 
## metadata(0):
## assays(1): counts
## rownames(5): H3 CD99 PIN CD8a CDH
## rowData names(0):
## colnames: NULL
## colData names(8): ImageNb object_id ... m.majoraxis m.eccentricity
## reducedDimNames(0):
## mainExpName: NULL
## altExpNames(0):

# Use cytoviewer with images, masks and object
app_1 <- cytoviewer(image = cur_images, 
                  mask = cur_masks, 
                  object = cur_sce, 
                  img_id = "ImageNb", 
                  cell_id = "object_id")

if (interactive()) {
  
  shiny::runApp(app_1, launch.browser = TRUE)

  }