SmartX LabHKUSTstRSC
GitHub

A noise-corrected metric for evaluating spatial transcriptomics predicted from histology

Jiarui Ouyang†, Yihang Gao†, Cheng Jin, Yihui Wang, Yingxue Xu, Ling Liang, Fengtao Zhou, Zhiyuan Cai and Hao ChenThe Hong Kong University of Science and Technology · † Equal contribution

Expression predicted from H&E-stained sections is usually scored by correlation over whole sections. That score rewards a model for recognizing broad tissue regions: in HER2ST, a class-mean baseline that gives every spot of a region the same value reaches Pearson r = 0.111, above the median of 11 published methods (0.108). At fine scales the error runs the other way, because sparse RNA counts carry sampling noise that no prediction can reproduce, and Pearson r with the measured RNA expression understates prediction quality.

The reproducible-signal correlation (RSC) splits the measured RNA molecules at random into two halves. Both halves share the expected expression and carry independent sampling noise. At each spatial scale, and within tissue regions or cell classes, RSC scores a prediction against the signal the two halves share: 1 when the prediction accounts for all reproducible variation at that scale, 0 for chance agreement. RSC is inspired by Fourier shell correlation in cryo-electron microscopy, which compares two half-maps of the same structure.

Calculate RSC on your dataResults
An ideal prediction, at each scaleFig. 1c, three colorectal cancer patients
0.40.60.81.0Correlation<128128–256256–5120.5–1k1–2k2–4k≥4kSpatial scale (µm)RSC at <128 µm: 0.997RSC at 128–256 µm: 0.996RSC at 256–512 µm: 0.996RSC at 0.5–1k µm: 0.996RSC at 1–2k µm: 0.996RSC at 2–4k µm: 0.996RSC at ≥4k µm: 0.996Pearson r with measured RNA expression at <128 µm: mean 0.516, patients 0.441 to 0.569Pearson r with measured RNA expression at 128–256 µm: mean 0.780, patients 0.704 to 0.847Pearson r with measured RNA expression at 256–512 µm: mean 0.921, patients 0.875 to 0.944Pearson r with measured RNA expression at 0.5–1k µm: mean 0.980, patients 0.965 to 0.993Pearson r with measured RNA expression at 1–2k µm: mean 0.995, patients 0.992 to 0.998Pearson r with measured RNA expression at 2–4k µm: mean 0.999, patients 0.997 to 0.999Pearson r with measured RNA expression at ≥4k µm: mean 1.000, patients 1.000 to 1.000RSC 0.997Pearson r 0.44–0.57

Even a prediction equal to the underlying expression reaches a Pearson r with measured RNA expression of only 0.44 to 0.57 at the finest scale. Its RSC there is 0.997.

Fewer molecules, the same predictionsExtended Data Fig. 1d, Ridge–Phikon-v2 in nine Xenium samples
0.20.30.40.50.6Correlation11/21/41/8Fraction of molecules keptWhole-section Pearson r, 1 of the molecules: 0.509Whole-section Pearson r, 1/2 of the molecules: 0.481Whole-section Pearson r, 1/4 of the molecules: 0.448Whole-section Pearson r, 1/8 of the molecules: 0.408Fine-scale RSC, 1 of the molecules: 0.337Fine-scale RSC, 1/2 of the molecules: 0.354Fine-scale RSC, 1/4 of the molecules: 0.369Fine-scale RSC, 1/8 of the molecules: 0.386Fine-scale split-half reliability, 1 of the molecules: 0.416Fine-scale split-half reliability, 1/2 of the molecules: 0.360Fine-scale split-half reliability, 1/4 of the molecules: 0.305Fine-scale split-half reliability, 1/8 of the molecules: 0.2470.509Whole-section Pearson r0.337Fine-scale RSC0.416Fine-scale split-halfreliability

Removing molecules at random lowers whole-section Pearson r from 0.509 to 0.408, while fine-scale RSC changes from 0.337 to 0.386.

All figures and their data in Results

How RSC is computed

  1. Split

    For each gene and each bin, spot or cell with count N, draw NA ~ Binomial(N, 0.5) and set NB = N − NA. The split is made on raw counts, before any normalization.

  2. Decompose

    Transform and center both halves, center and scale the prediction by gene, and take all three apart into spatial scales: a Fourier basis on a regular grid, the eigenvectors of a neighbor-graph Laplacian on spots, diffusion operators on a cell graph.

  3. Correlate

    At each scale b, sum over all genes and locations the mean covariance C of the prediction with the two halves, the variance Vmodel of the prediction and the covariance VRNA between the halves. Repeat over independent splits and average.

RSCb = C / √(Vmodel · VRNA)

VRNA estimates the variance of the reproducible signal. The correlation between the two halves, VRNA/√(VAVB), is the split-half reliability ρb. For halves of equal variance, the Pearson r of a prediction with the mean of the halves is RSCb · √(2ρb/(1 + ρb)), so it falls wherever counts are sparse even when the prediction is right. More in the user guide

Figure 1 of the manuscript: an overview of RSC (random split of the measured counts, decomposition into broad, mid and fine scales, and correlation of the prediction with each half), whole-section correlation of a class-mean baseline and 11 published methods in HER2ST, RSC and Pearson r with measured RNA expression for an ideal prediction across spatial scales, and a benchmark summary for spatial scales, tissue regions, cell classes and biological analyses.
Fig. 1 | Reproducible-signal correlation (RSC) avoids overstating and understating prediction quality. a, A random split partitions the measured transcripts into two halves with independent Poisson noise (1), a multiscale decomposition isolates broad, mid and fine components (2), and the correlation of the prediction with each half is divided by the square root of the split-half reliability (3). b, Along a transect across tumor and non-tumor regions, and in whole-section Pearson r over 785 genes in HER2ST (8 patients), a class-mean baseline with no variation within regions scores above the median of 11 published methods (r = 0.111 versus 0.108). c, For a prediction equal to the underlying expression in three colorectal cancer patients, Pearson r with the measured RNA expression drops at fine scales, whereas RSC stays near 1 from below 128 µm to 4,096 µm and above. d, Spot-level predictions fall to an RSC of 0.298 at the finest scale despite high broad-scale RSC (Fig. 3); normalized pairwise accuracy drops toward the majority-order baseline once spot pairs are matched or evaluated within regions (Fig. 2); fine-scale RSC within malignant cells separates the single-cell methods (Fig. 4); models capture differences between regions but miss the variation within TLS, and single-cell methods differ in the enrichment of proliferation hotspots (Figs. 4e and 5f).

Results

  1. Fig. 2

    Tissue regions

    Normalized pairwise accuracy of 11 methods in HER2ST: 28.5% for unmatched spot pairs, 6.7% for pairs matched for total transcript count and expression profile, and 6.6% for matched pairs in the same annotated region. Ranks by whole-section correlation agree with ranks on unmatched pairs (Kendall's τb = 0.818) but hardly with ranks within regions (τb = 0.091).

    Fig. 2 in Results
  2. Fig. 3

    Spatial scales

    RSC of Ridge–Phikon predictions in colorectal tissue: 0.800 at scales of 4,096 µm and above, 0.298 below 128 µm. In nine Xenium samples of breast, pancreatic and lung cancer, 10 of 11 spot-level models have lower RSC at the finest than at the broadest scale in every sample. Features from pathology foundation models raise RSC at every scale; with each of the five image encoders, RSC drops by 0.31 to 0.34 from the broadest to the finest scale.

    Fig. 3 in Results
  3. Fig. 4

    Cell classes

    Fine-scale RSC within malignant cells, with a split-half reliability of 0.173: 0.242 for GHIST, 0.042 for DeepSpot2Cell and 0.020 for sCellST. Averaging the same predictions into 256-µm bins raises fine-scale RSC across all cells by 0.14 to 0.17. Measured enrichment in predicted proliferation hotspots: 0.335 for GHIST, 0.101 for sCellST and −0.064 for DeepSpot2Cell.

    Fig. 4 in Results
  4. Fig. 5

    Tertiary lymphoid structures

    Eight TLS genes in two kidney and two lung sections: Ridge–Phikon-v2 predictions separate TLS from other spots with a mean AUROC of 0.93 to 0.99, measured expression of single spots with 0.61 to 0.71. Their RSC is 0.59 to 0.79 between regions but 0.25 to 0.29 within TLS.

    Fig. 5 in Results
  5. Fig. 6

    Pairwise difference loss

    Fine-scale RSC of 11 spot-level models in nine Xenium samples: 0.194 after standard training and 0.205 with the loss. On neighboring spots the loss raises fine-scale RSC by 0.009 more than on the same number of non-neighboring pairs.

    Fig. 6 in Results

Your own predictions