topic


This paper introduces ResNet-L2 (RL2), a novel metric for evaluating the quality of generative models and images in histopathology, addressing the limitations of traditional metrics like FID, especially in data-scarce scenarios.


coremsg

EVALUATION METRIC FOR QUALITY CONTROL AND
GENERATIVE MODELS IN HISTOPATHOLOGY IMAGES
Pranav Jeevan∗
Neeraj Nixon∗
Abhijeet Patil
Amit Sethi
Department of Electrical Engineering
Indian Institute of Technology Bombay
Mumbai, India
{pjeevan, 20d070056, abhijeetptl, asethi}@iitb.ac.in
ABSTRACT
Our study introduces ResNet-L2 (RL2), a novel metric for evaluating generative models and image
quality in histopathology, addressing limitations of traditional metrics, such as Fréchet inception
distance (FID), when the data is scarce. RL2 leverages ResNet features with a normalizing flow to
calculate RMSE distance in the latent space, providing reliable assessments across diverse histopathol-
ogy datasets. We evaluated the performance of RL2 on degradation types, such as blur, Gaussian
noise, salt-and-pepper noise, and rectangular patches, as well as diffusion processes. RL2’s monotonic
response to increasing degradation makes it well-suited for models that assess image quality, proving
a valuable advancement for evaluating image generation techniques in histopathology. It can also be
used to discard low-quality patches while sampling from a whole slide image. It is also significantly
lighter and faster compared to traditional metrics and requires fewer images to give stable metric
value.
Keywords Evaluation metrics · Normalizing flows · Generative models · Diffusion · L2
1
Introduction and Background
Generative models, particularly generative adversarial networks (GAN), variational autoencoders (VAE), and diffusion
are used in histopathology for various tasks, such as data augmentation [1], anomaly detection [2], and synthetic image
generation [3], for improving diagnostic models by addressing scarcity of data. GANs and diffusion generate realistic
images to train models for tasks, such as cancer detection [4], while VAEs aid in anomaly detection by reconstructing
healthy tissues and identifying deviations [5]. These models also facilitate style transfer between staining protocols,
thus reducing costs and enhancing cross-modal comparisons. Overall, generative models enhance data efficiency and
diagnostic accuracy in histopathology.
Evaluating the quality and performance of generative models is challenging due to the lack of well-established, theory-
backed metrics. This complexity stems from the need to assess various aspects of generated images, such as quality,
aesthetics, realism, and diversity, all of which are inherently subjective. While human evaluation is effective, it is costly,
time-consuming, and impractical for large datasets. As a result, researchers rely on automated evaluation methods that
should ideally respond consistently to image degradation, accurately measure image quality, and remain computationally
efficient, particularly in resource-limited environments.
Commonly used metrics for evaluating generative models include Inception Score (IS) [6], Kernel Inception Distance
(KID) [7], Fréchet Inception Distance (FID) [8], and contrastive language-image pre-training (CLIP) Maximum Mean
Discrepancy (CMMD) [9]. KID, FID and CMMD require thousands of real and generated images to compute a stable
metric which becomes a big issue in medical domain with data scarcity. Flow-based Likelihood Distance [10] have
been proposed as an alternative, functioning effectively with fewer data, but they come with the complexity of training
large normalizing flows, which is inefficient due to the need to maintain the large latent space. Another limitation of
these metrics is their reliance on Inception-V3 features, which are trained on natural image datasets such as ImageNet.
∗Equal Contribution.
arXiv:2411.01034v1  [eess.IV]  1 Nov 2024

ResNet-L2
Real Images
ResNet-18
Normalizing Flow
N
Real Images
Generated
 Images
Training
Evaluation
L2
ResNet-18
Normalizing Flow
N
ResNet-18
Normalizing Flow
N
RL2
Avg
Avg
Figure 1: The process of computing RL2. In the training phase, the ResNet-normalizing flow network is trained on the
given real (high-quality) images. In the evaluation phase, real and generated (or evaluation) images are passed through
the network, and the L2 distance between mean of latent vectors of real and generated (or evaluation) images is used as
the final metric.
While effective for evaluating generative models that produce natural images, this approach performs poorly in domains,
such as medical imaging, particularly in histopathology, where the image characteristics differ significantly from natural
images.
Additionally, a quality metric trained on real patches of high quality can also be used to filter out low-quality patches
from a whole slide image while training and testing deep learning pipelines for weakly supervised learning in histopathol-
ogy [11].
2
ResNet-L2
Normalizing flows are the only class of generative models capable of providing exact and efficient likelihood estimates
for data [12]. However, they require the latent space to have the same dimensionality as the input data, leading to
substantial computational demands, particularly for high-dimensional inputs, such as images. To address this issue,
we apply normalizing flows to the extracted image features rather than the raw images, thus effectively reducing
the dimensionality and computational complexity while preserving the model’s ability to estimate data likelihoods
accurately and efficiently.
Our proposed ResNet-L2 (RL2) metric employs a normalizing flow applied to image features extracted from a pre-
trained ResNet-18, and computes the root mean square distance between the latent vectors of real and generated images.
Specifically, we start with a set of real images, denoted as R, and a set of generated images, G as shown in Figure 1.
We use a pre-trained ResNet-18, truncated before the final layer, as the feature extractor f, and a normalizing flow N,
parameterized by two trainable sets of parameters, θ and ξ respectively.
xf = f(xin, θ)
xin ∈R, xf ∈R512
(1)
z = N(xf, ξ)
z ∈R512, xf ∈R512
(2)
We train the combined network (Eq. 1 and Eq. 2) on real images R to maximize the log-likelihood of the real data.
This training process is performed only once, allowing the network to map image features into a Gaussian latent
space that assigns high likelihoods to real image features. Notably, since ResNet-18 is a significantly lighter model
2

ResNet-L2
(11 M parameters) compared to Inception-V3 (23 M parameters) or CLIP, which are used in previous metrics, our
approach offers faster and more computationally efficient metric computation. Moreover, by applying the normalizing
flow to the extracted image features rather than the raw images, we further enhance efficiency due to the reduced
dimensionality of the latent space, enabling quicker training. Training on real histopathology data allows the model to
capture domain-specific features and assign higher probabilities to real histopathology images. This domain-specific
adaptation is absent in previous metrics, which rely solely on frozen networks pre-trained on ImageNet.
After training the network on real images R, we extract the mean of the latent vectors z for real and generated images
and compute the L2 (Euclidean distance) between the real and generated latent vectors2.
zr =
P
z∈R z
|R|
zg =
P
z∈G z
|G|
z ∈R512
(3)
RL2 = ∥zr −zg∥2 =
v
u
u
t
n
X
i=1
(zri −zgi)2
(4)
The computation of L2 distance between two vectors is computationally lighter and faster than the computing Fréchet
distance or maximum mean discrepancy between them. Hence, RL2 is lighter, faster and efficient to compute than
previous metrics.
z-level
RL2
0.00
0.25
0.50
0.75
1.00
1.25
-8
-6
-4
-2
0
2
4
6
8
Figure 2: The behaviour of RL2 shows that it is monotonic with increasing levels of blur caused by variation of z-levels
in FocusPath dataset. The top row shows the RL2 value and the bottom row shows the z-values.
3
Datasets, Implementation and Experiments
The FocusPath dataset [13] consists of 864 image patches, each with a resolution of 1024×1024 pixels in sRGB format,
capturing varying degrees of focus. These patches are cropped from nine distinct whole slide images (WSIs), with
2Code available at https://github.com/pranavphoenix/RL2
3

ResNet-L2
Salt and Pepper Noise
RL2
0
2
4
6
8
10
0.001
0.002
0.004
0.006
0.008
0.01
Figure 3: The behaviour of RL2 shows that it is monotonic with increasing levels of salt and pepper noise in
histopathology images from 0 z-level FocusPath dataset.. The top row shows the RL2 value and the bottom row shows
the noise values.
16 different z-levels employed to simulate various out-of-focus conditions. The tissue slides selected for the dataset
feature diverse color staining techniques, ensuring that the dataset reflects a wide range of histopathological staining
variations. The dataset involves image patches with varying degrees of focus, translating to different levels of blur. This
dataset serves as an excellent tool for assessing our metric’s capability to distinguish between blurry and clear images.
Moreover, it allows us to evaluate the monotonicity of our metric with respect to the degree of blur, as it should assign
higher values to images with greater blurriness. We train our network only on images with z-level 0 (which are blur free)
and evalaute it on all the other images with higher degree of blur. We add salt and pepper noise and rectangular patch
noise to the 0 z-level images for our experiments involving other kinds of noise. We used images of 100× resolution
from BreakHis [14] dataset for generating synthetic images for our experiments to test RL2 for diffusion noise.
For the experiment to filter out low-quality noisy image patches from good quality ones, we used HistoROI dataset [11].
HistoROI dataset is developed to segment WSIs into six key classes: epithelium, stroma, lymphocytes, adipose, artifacts,
and miscellaneous. Artifacts in this dataset include out-of-focus areas, tissue folds, cover slips, air bubbles, pen marks,
and areas with extreme over- or under-staining, which were carefully labeled to aid in quality control for pathology
image analysis. We trained our network using 1500 artifact-free patches from epithelium and stroma classes. We then
use this network to classify 474 artifact patches from 474 clean patches.
We use a normalizing flow using real-valued non-volume preserving (real NVP) transformations [15] in our experiments.
We utilized 256 × 256 patches from all the datasets, achieved by cropping the original images. All experiments were
run on Nvidia A6000 GPU. For experiments using salt and pepper noise, rectangular patch noise, and diffusion noise,
we used the same experimental setup used by FID paper [8].
4
Results and Discussion
Figure 2 illustrates that as the level of blur increases, indicated by higher z-levels, ranging from 1 to 8 and −1 to −8,
our metric RL2 effectively identifies these images as noisier compared to the 0 z-level images. Moreover, RL2 displays
a monotonous increase with greater intensity of blur in both directions, highlighting its effectiveness as an ideal tool for
identifying blurry images compared to clear, noise-free ones.
4

ResNet-L2
Rectangular Patch Size
RL2
0.0
0.5
1.0
1.5
2.0
10
20
40
60
80
100
Figure 4: The behaviour of RL2 shows that it is monotonic with increasing levels of rectangular patch noise in
histopathology images from 0 z-level FocusPath dataset.. The top row shows the RL2 value and the bottom row shows
the noise values.
Diffusion steps
RL2
0.5
1.0
1.5
2.0
900
925
950
975
1000
Figure 5: The behaviour of RL2 shows that it is monotonic with increasing levels of salt and pepper noise in
histopathology images from 0 z-level FocusPath dataset.
A robust evaluation metric for generative models must effectively distinguish between varying levels of different
noise types. In our experiments with salt-and-pepper noise and rectangular patch noises, common in histopathology
images, our metric, RL2, shows a monotonic increase with rising noise levels as shown in Figure 3 and Figure 4. This
demonstrates its effectiveness in evaluating image quality. The capability to identify corruptions in real images makes
our metric a valuable tool for detecting subtle differences caused by various noises. Even when synthetic image data
significantly differs from real data, our metric reliably identifies varying levels of various noise.
5

ResNet-L2
The results of our evaluation metric on the diffusion process, are illustrated in Figure 5. As shown, an increase in the
number of diffusion steps correlates with a reduction in noise levels, a pattern our metric effectively captures through
lower values. This monotonic reduction highlights quality improvements in the diffusion process, essential for accurate
evaluation of histopathological images from generative models.
In the experiment for filtering out low-quality patches with artifacts from high quality patches, our network trained only
on clean patches gives an AUC score of 0.76. Even without being trained on the artifacts, the model is able to identify
clean patches from noisy ones based on the their likelihood.
RL2 metric achieves stability with just 300 samples compared to traditional methods which FLD required over 20,000
images to reach a stable value [9]. This significant sample efficiency of RL2 enables it to produce reliable, stable metric
values with minimal data.
5
Conclusions
Popular generative model evaluation metrics, such as FID, have proven unreliable for assessing the latest models,
such as diffusion. This is particularly true in domains, such as histopathology, where the available images are limited.
Previous metrics requiring thousands of samples for stability is not suitable, nor are those metrics relying on pre-trained
ImageNet weights due to the distinct nature of histopathology image features.
We developed a new evaluation metric based on normalizing flow and measures the L2 distances between real
and generated features. Our metric has been shown to be monotonic with respect to various noise types found in
histopathology, including noise, occlusion, and blur. Additionally, because our metric is lighter, faster and requires
significantly fewer resources, it is efficient than previous metrics. We believe this work will empower researchers to
train and test generative models more efficiently.
6
Compliance with Ethical Standards
This research used public data, ethical approval was not required as confirmed by the license attached with the data.
References
[1] Xianchao Guan, Yifeng Wang, Yiyang Lin, and Yongbing Zhang. Data augmentation based on discrimdiff
for histopathology image classification. In Data Augmentation, Labelling, and Imperfections: Third MICCAI
Workshop, DALI 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 12, 2023,
Proceedings, page 53–62, Berlin, Heidelberg, 2024. Springer-Verlag.
[2] Hristina Uzunova, Sandra Schultz, Heinz Handels, and Jan Ehrhardt. Unsupervised pathology detection in medical
images using conditional variational autoencoders. International Journal of Computer Assisted Radiology and
Surgery (Int J CARS), 14:451–461, 2019.
[3] Srikar Yellapragada, Alexandros Graikos, Prateek Prasanna, Tahsin Kurc, Joel Saltz, and Dimitris Samaras.
Pathldm: Text conditioned latent diffusion model for histopathology, 2023.
[4] Wenyuan Li, Jiayun Li, Jennifer Polson, Zichen Wang, William Speier, and Corey Arnold. High resolution
histopathology image generation and segmentation through adversarial training.
Medical Image Analysis,
75:102251, 2022.
[5] Harsha V. Guleria, Ahmed M. Luqmani, Harshal D. Kothari, Pranay Phukan, Shubham Patil, Pranjal Pareek,
Ketan Kotecha, Ajith Abraham, and Luma A. Gabralla. Enhancing the breast histopathology image analysis for
cancer detection using variational autoencoder. International Journal of Environmental Research and Public
Health, 20(5):4244, 2023.
[6] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved
techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in
Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[7] Mikołaj Bi´nkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In
International Conference on Learning Representations, 2018.
[8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a
two time-scale update rule converge to a local nash equilibrium, 2018.
6

ResNet-L2
[9] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar.
Rethinking fid: Towards a better evaluation metric for image generation, 2024.
[10] Pranav Jeevan, Neeraj Nixon, and Amit Sethi. Normalizing flow-based metric for image generation, 2024.
[11] Abhijeet Patil, Harsh Diwakar, Jay Sawant, Nikhil Cherian Kurian, Subhash Yadav, Swapnil Rane, Tripti Bameta,
and Amit Sethi. Efficient quality control of whole slide pathology images with human-in-the-loop training.
Journal of Pathology Informatics, 14:100306, 2023.
[12] Simon J.D. Prince. Understanding Deep Learning. The MIT Press, 2023.
[13] M. S. Hosseini, Y. Zhang, and K. N. Plataniotis. Encoding visual sensitivity by maxpol convolution filters for
image sharpness assessment. IEEE Transactions on Image Processing, 28(9):4510–4525, Sep. 2019.
[14] Fabio A. Spanhol, Luiz S. Oliveira, Caroline Petitjean, and Laurent Heutte. A dataset for breast cancer histopatho-
logical image classification. IEEE Transactions on Biomedical Engineering, 63(7):1455–1462, 2016.
[15] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp, 2017.
7



Evaluation Metric for Quality Control and Generative Models in Histopathology Images

### title_rewrite
ResNet-L2: A Novel and Efficient Metric for Evaluating Generative Models and Image Quality in Histopathology Images

### category
Computer Vision

### topic
Image Quality Assessment

### coremsg
This paper introduces ResNet-L2 (RL2), a novel metric for evaluating the quality of generative models and images in histopathology, addressing the limitations of traditional metrics like FID, especially in data-scarce scenarios.

### note
## ResNet-L2: A Novel and Efficient Metric for Evaluating Generative Models and Image Quality in Histopathology Images

This research paper proposes a new metric called ResNet-L2 (RL2) for evaluating the quality of images generated by generative models in the field of histopathology. The authors argue that existing metrics like FID are inadequate for this task, particularly when dealing with limited data, a common challenge in medical imaging.

### Bibliographic Information:
Pranav Jeevan, Neeraj Nixon, Abhijeet Patil, & Amit Sethi. (2024). Evaluation Metric for Quality Control and Generative Models in Histopathology Images. arXiv:2411.01034v1 [eess.IV].

### Research Objective:
The study aims to develop a more reliable and efficient metric for evaluating generative models and image quality in histopathology, addressing the limitations of traditional metrics in handling data scarcity and domain-specific features.

### Methodology:
The researchers propose RL2, which leverages a pre-trained ResNet-18 model for feature extraction and a normalizing flow to map these features into a Gaussian latent space. The L2 distance between the mean latent vectors of real and generated images serves as the quality metric. The method is evaluated on various histopathology datasets with different types of image degradation, including blur, noise, and artifacts.

### Key Findings:
RL2 demonstrates superior performance compared to traditional metrics, exhibiting monotonicity with increasing levels of image degradation and achieving stability with significantly fewer samples. The metric effectively distinguishes between varying levels of noise and blur, proving valuable for assessing image quality and identifying corrupted images.

### Main Conclusions:
RL2 offers a robust and efficient alternative for evaluating generative models and image quality in histopathology, particularly in data-limited settings. Its ability to capture domain-specific features and handle various noise types makes it a valuable tool for researchers in the field.

### Significance:
This research contributes significantly to the advancement of image quality assessment in histopathology, providing a more reliable and efficient metric for evaluating generative models and facilitating the development of improved diagnostic and analytical tools.

### Limitations and Future Research:
While RL2 shows promise, further investigation into its generalizability across different histopathology datasets and generative model architectures is warranted. Exploring the integration of RL2 with other quality metrics could further enhance its robustness and applicability.

### data_sheet
- RL2 metric achieves stability with just 300 samples.
- Traditional methods like FLD require over 20,000 images to reach a stable value.

### quotes
- "Normalizing flows are the only class of generative models capable of providing exact and efficient likelihood estimates for data [12]."
- "Popular generative model evaluation metrics, such as FID, have proven unreliable for assessing the latest models, such as diffusion."

### further_questions
- How might the principles behind RL2 be applied to other medical imaging modalities beyond histopathology?
- Could the reliance on a pre-trained ResNet-18 model introduce biases towards certain image features, and how can these biases be mitigated?
- If artistic expression can be evaluated through metrics like "emotional impact" or "compositional balance," could similar qualitative metrics be developed for scientific imaging to assess aspects beyond technical quality? 


Image Quality Assessment

resnet-l2-a-novel-and-efficient-metric-for-evaluating-generative-models-and-image-quality-in-histopathology-images

note


The study aims to develop a more reliable and efficient metric for evaluating generative models and image quality in histopathology, addressing the limitations of traditional metrics in handling data scarcity and domain-specific features.


Research Objective:


Pranav Jeevan, Neeraj Nixon, Abhijeet Patil, & Amit Sethi. (2024). Evaluation Metric for Quality Control and Generative Models in Histopathology Images. arXiv:2411.01034v1 [eess.IV].


Bibliographic Information:


ResNet-L2: A Novel and Efficient Metric for Evaluating Generative Models and Image Quality in Histopathology Images


ResNet-L2: A Novel and Efficient Metric for Evaluating Generative Models and Image Quality in Histopathology Images

ResNet-L2: A Novel and Efficient Metric for Evaluating Generative Models and Image Quality in Histopathology Images

Mukauta tiivistelmää

Kirjoita tekoälyn avulla

Luo viitteet

Käännä lähde

Luo miellekartta

Siirry lähteeseen

Evaluation Metric for Quality Control and Generative Models in Histopathology Images

Hae PDF-tiivistelmä sekunneissa