34 Deep learning
34.1 Preamble
34.1.1 Introduction
Deep learning refers to a family of machine learning methods based on artificial neural networks with multiple (i.e., “deep”) layers, capable of learning hierarchical representations directly from raw data (e.g., pixels, sequences, graphs) rather than from hand-crafted features. (LeCun, Bengio, and Hinton 2015) Several architectural families recur throughout single-cell and spatial omics analysis:
Convolutional neural networks (CNNs) apply learned filters that slide across an image to detect local patterns (edges, textures, nuclei shapes); they are the workhorse behind most cell/nucleus segmentation and image classification tools (Chapter 33).
Graph neural networks (GNNs) operate on data structured as graphs (e.g., cells connected to their spatial neighbors) and propagate information along edges, making them a natural fit for modeling spatial neighborhoods and cell-cell interactions (Chapter 29, Chapter 22).
Transformers rely on self-attention to weigh the relevance of every element of an input (e.g., a gene, a token, an image patch) with respect to every other element, without the locality constraints of CNNs. Transformers underlie most large language models (LLMs) as well as the majority of recent foundation models discussed below, including for non-text data such as single-cell profiles or histology images.
Large language models (LLMs) are transformer-based models trained on massive text corpora; while not designed for omics data, the same self-attention machinery has been repurposed to model genes, cells, or spatial spots as “tokens” (see
CellPLMbelow).
Taken together, these architectures increasingly converge in foundation models (FMs), which extend the same principles to (pre-)training at scale. The Stanford Institute for Human-Centered Artificial Intelligence describes an FM, in essence, as a model trained on broad data (typically via self-supervision at scale) that can subsequently be adapted (e.g., fine-tuned) to a wide variety of downstream tasks. (Bommasani et al. 2022)
“Self-supervision” here means that the model learns from unlabeled data by solving a proxy task derived from the data itself, without requiring manual annotation. Common strategies include masking, where part of the input (such as a gene, a patch, or a token) is hidden and the model is trained to reconstruct it from context, and contrastive learning (training the model to recognize when two inputs, e.g., two augmented views of the same image patch, represent the same underlying entity, as used by DINOv2, discussed further below for KRONOS). This is what allows FMs to be pre-trained on very large collections of unlabeled cells or images before being adapted to specific, labeled downstream tasks. (Gui et al. 2024)
FMs are typically trained on very large datasets and have high complexity (commonly billions of parameters), requiring multiple GPUs and long, costly training runs. Once trained, however, they can be reused “off-the-shelf” via fine-tuning or simply by extracting embeddings, without repeating the expensive pre-training step. Unlike task-specific models, FMs can in principle be applied to any data type: text, images, or tabular/matrix data such as count matrices, and in spatial biology, they offer a natural way to integrate histology (H&E) or immunofluorescence (IF) images with spatially resolved molecular data.
34.2 Background
Historically, most computational methods for single-cell and spatial omics were purpose-built for one task and one data modality: a segmentation model trained on one imaging platform, a clustering method tuned to one assay, and so on. FMs shift this paradigm by decoupling representation learning (learning a general-purpose embedding of cells, images, or genes) from the downstream task (classification, clustering, spatial domain assignment, etc.). The same pre-trained backbone can then be reused across studies, platforms, and tasks, at the cost of requiring careful evaluation of whether learned representations actually transfer well to a given downstream setting (see Outlook below).
34.3 Foundation models
Foundation models (FMs) represent a modern paradigm where massive deep learning architectures (e.g., transformers) are pre-trained on millions of single-cell profiles, biological images, or spatial omics profiles, to learn generalizable representations. Nearly all of these models are developed and distributed in Python; R infrastructure to interface with pre-trained models (making use of model weights and embeddings for downstream analysis) is actively being developed, as discussed below. The remainder of this section first outlines how a pre-trained FM is typically adapted to a new task, then walks through representative models organized by data modality.
| Model | Input | Architecture | Output |
|---|---|---|---|
| Prov-GigaPath (Xu et al. 2024) | H&E | ViT | tile/slide embedding |
| UNI (Chen et al. 2024) | H&E | ViT | embedding |
| CONCH (Lu et al. 2024) | image + text | vision-language | embedding + zero-shot |
| KRONOS (Shaban et al. 2025) | IF | ViT (DINOv2-based) | patch embedding |
| Nimbus (Rumberger et al. 2025) | IF | marker-agnostic CNN | per-marker positive/negative |
| Geneformer (Theodoris et al. 2023) | scRNA | Transformer | cell embedding |
| scGPT (Cui et al. 2024) | scRNA | GPT | cell embedding |
| scFoundation (Hao et al. 2024) | scRNA | Transformer | cell embedding |
| Novae (Blampey et al. 2025) | spatial | GNN | domain/niche embedding |
| Nicheformer (Tejada-Lapuerta et al. 2025) | scRNA + spatial | Transformer | niche embedding |
34.3.1 Adapting a pre-trained FM to a new task
Having a pre-trained FM available does not by itself solve a downstream analysis task; the model still needs to be adapted, and there are three common strategies, in increasing order of computational cost:
Zero-shot inference uses the pre-trained model as-is, with no further training, either by extracting embeddings directly for tasks like clustering or nearest-neighbor search, or, for vision-language models such as CONCH, by prompting the model with natural-language class descriptions.
Linear probing keeps the pre-trained weights frozen and trains only a small linear (or otherwise simple) classifier/regressor on top of the extracted embeddings for a specific labeled task, a comparatively cheap strategy often used to benchmark how informative a given FM’s embeddings are.
Fine-tuning updates some or all of the pre-trained weights on task-specific labeled data. This typically yields the best task-specific performance but requires more labeled data and compute, and risks overfitting on small cohorts.
Which strategy is appropriate depends on how much labeled data is available for the task at hand and how similar that task is to the data the FM was pre-trained on; benchmarking more than one strategy is generally advisable before committing to FM-derived embeddings for a downstream analysis.
34.3.2 FMs for histology (H&E) images
In spatial transcriptomics, FMs pre-trained on hematoxylin and eosin (H&E) whole-slide images provide embeddings that summarize morphology at the patch, cell, or slide level, which can then be linked to gene expression, clinical outcomes, or spatial domains. Prominent examples include:
Prov-GigaPath (Xu et al. 2024), a vision transformer pre-trained on over a billion tiles from more than 170,000 whole-slide images, combining tile-level self-supervised pre-training with slide-level aggregation to capture both local and long-range tissue context.
UNI (Chen et al. 2024), a general-purpose pathology encoder trained via self-supervision on over 100 million histopathology image patches, designed to transfer well across a broad range of diagnostic tasks.
CONCH (Lu et al. 2024), a vision-language foundation model (not graph-based) trained on more than a million histology image-caption pairs, allowing it to be queried with text as well as images (e.g., zero-shot classification from natural-language prompts).
Since these models are typically compared on how well their embeddings predict gene expression from morphology, it is worth also mentioning HEST-1k (Jaume et al. 2024), a dataset and companion benchmark (the HEST-Benchmark) of over a thousand paired H&E whole-slide images and spatial transcriptomics profiles spanning multiple organs, species, and cancer types. Rather than an FM itself, HEST-1k provides a standardized way to evaluate and compare H&E foundation models (including Prov-GigaPath, UNI, and CONCH) on the task of predicting gene expression directly from histology.
34.3.3 FMs for multiplexed / immunofluorescence (IF) images
For highly multiplexed imaging (e.g., IMC, CODEX, CosMx protein panels), FMs face an additional challenge: the number and identity of channels/markers varies across experiments and platforms. Two representative approaches are:
KRONOS (Shaban et al. 2025), a foundation model purpose-built for spatial proteomics. It was trained on more than 47 million single-marker image patches spanning 175 protein markers and 16 tissue types, using a Vision Transformer (based on DINOv2) adapted to jointly encode multiple, heterogeneous markers. KRONOS extracts 384-dimensional embeddings per image patch, usable for cell phenotyping, artifact detection, tissue-region classification, or outcome prediction.
Nimbus (Rumberger et al. 2025), a model that operates marker-by-marker (rather than jointly across all channels) to classify, for each cell, whether a given marker is positive or negative directly from raw pixel values rather than integrated intensities. Because it treats each marker independently, Nimbus is portable across multiplexed-imaging platforms with different channel counts and panels, without retraining.
34.3.4 FMs for single-cell omics
Several FMs treat single-cell (and, by extension, spot- or bin-level spatial) gene expression profiles analogously to text, with genes or cells playing the role of “tokens” and “sentences”:
Geneformer (Theodoris et al. 2023), a transformer pre-trained on ~30 million single-cell transcriptomes, representing each cell as a ranked sequence of gene tokens.
scGPT (Cui et al. 2024), a generative pre-trained transformer for single-cell biology trained across tens of millions of cells; a spatially-aware extension (
scGPT-spatial) has been proposed but is not yet broadly integrated into spatial workflows.scFoundation (Hao et al. 2024), a large-scale model covering close to the full human transcriptome (~20,000 genes) as input, rather than a reduced gene set.
Not an FM in the strict sense, but conceptually related and worth mentioning: CellPLM (Wen et al. 2023) is a pre-trained “cell language model” with cells as tokens and tissues as sentences-an idea directly inspired by large language models. It supports several downstream tasks including denoising and annotation of scRNA-seq data, imputation of ST data, and perturbation-response prediction.
34.3.5 FMs for spatial transcriptomics
Although the models above were largely developed for non-spatial single-cell data, spatial transcriptomics profiles can often be passed through them for comparison with spatial-specific models, even without native support for spatial coordinates. Models developed specifically with spatial context in mind include:
Novae (Blampey et al. 2025), a graph-based deep learning model for spatial domain and niche assignment across samples and technologies, discussed further in Chapter 29.
Nicheformer (Tejada-Lapuerta et al. 2025), a transformer-based FM jointly pre-trained on over 110 million dissociated single-cell and spatially resolved cells across 73 tissues. By encoding modality, organism, and assay as explicit tokens, Nicheformer learns a shared representation of single-cell and spatial data, allowing spatial context (e.g., niche or neighborhood composition) learned from spatial assays to be transferred onto dissociated scRNA-seq data that was never spatially profiled.
For spatial omics more broadly, FM-derived embeddings increasingly overlap with spatial domain and neighborhood analysis, since they provide representations that can be clustered, compared across samples, or used as input to downstream analyses; see Chapter 29 and Chapter 22 for representative domain- and niche-level workflows.
34.3.6 FMs in the Bioconductor ecosystem
Because virtually all FMs above are implemented and distributed in Python, using them from R currently relies on Python interoperability layers such as reticulate and basilisk (Chapter 8). Community efforts are underway to bring foundation-model functionality natively into the Bioconductor ecosystem, providing R users with wrappers to run pre-trained models and retrieve their embeddings for downstream analysis without having to write custom Python interoperability code for each model. The BiocFM working group collects these ongoing efforts and is a useful entry point for tracking newly available R interfaces to FMs as this space matures.
34.3.7 Practical considerations: access, licensing, and compute
Beyond conceptual differences, FMs vary considerably in how easy they are to actually obtain and run, which is often the first practical hurdle for a new user:
Access and licensing. Many histology FMs are gated: for example, Prov-GigaPath and UNI are distributed through Hugging Face and require requesting access and agreeing to a data use agreement (typically restricting use to non-commercial research) before the weights can be downloaded, whereas CONCH’s weights are comparatively more openly accessible. Always check a model’s license and any usage restrictions before relying on it, particularly for clinical or commercial applications.
Compute requirements. Pre-trained checkpoints for these models are often several hundred megabytes to a few gigabytes in size, and while extracting embeddings from a pre-trained model (“inference”) is far cheaper than pre-training, it typically still benefits from, or requires, a GPU for reasonable runtimes on large whole-slide images or large cell atlases; CPU- only inference is usually possible but considerably slower. Fine-tuning, in particular, generally requires GPU resources.
Reproducibility. Because most FMs evolve quickly (new checkpoints, architecture variants), it is good practice to record the exact model version/checkpoint used, in addition to the usual software versions, when reporting FM-based results.
34.3.8 When should I use a foundation model?
Given the added complexity of FMs relative to simpler, task-specific methods, it is worth pausing before defaulting to one. As a rule of thumb:
Use one when annotated data are scarce. FMs are most valuable when you have little or no labeled data for your specific task, since their pre-trained representations can substitute for large labeled training sets.
Use one when transfer learning is desirable. If you need to compare or combine data across studies, platforms, or technologies, FM embeddings provide a shared representation space that a bespoke, dataset-specific model would not.
Benchmark against simpler baselines. As discussed in Outlook, FMs do not always outperform linear models or PCA on a given task-always compare against a simple baseline before committing to an FM-based pipeline.
Verify that your tissue, species, and platform are represented in pre-training. FM performance degrades outside the domain(s) seen during pre-training; check the original publication’s training data before assuming embeddings will transfer to your data.
Record the checkpoint and version used. FMs are updated frequently; note the exact model checkpoint/version alongside your usual software versions to keep results reproducible.
34.3.9 Outlook
FM-derived embeddings are not a universal replacement for simpler methods, and their added complexity should be weighed against the task at hand. Two cautionary results are worth highlighting: for the prediction of molecular changes upon perturbation (e.g., disease state, treatment response), deep-learning-based approaches have been shown not to (yet) outperform simple linear baselines (Ahlmann-Eltze, Huber, and Anders 2025); similarly, Kedzierska et al. (2025) report out-performance by simpler methods in the context of cell type annotation. As with any pre-trained model, it is important to check whether the domains (tissue types, platforms, species) seen during pre-training are representative of the data at hand before relying on FM embeddings for a new task.