Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration
Keywords: Graph Convolutional Networks, Multi-Label Chest X-ray, image classification, Graph Representation
Embedded Files
0. Motivation, Objective and Related Works:
2. C5_Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration_MICCAI_2020 [Paper] [Code] [Poster] [Slides] [Graphical abstract] [Talk (YouTube, YouKu)]
Motivation:
Medical images are naturally associated with rich semantics about the human anatomy, reflected in an abundance of recurring anatomical patterns, offering unique potential to foster deep semantic representation learning and yield semantically more powerful models for different medical applications.
But how exactly such strong yet free semantics embedded in medical images can be harnessed for self-supervised learning remains largely unexplored.
Objective:
To this end, we train deep models to learn semantically enriched visual representation by self-discovery, selfclassification, and self-restoration of the anatomy underneath medical images, resulting in a semantics-enriched, general-purpose, pre-trained 3D model, named Semantic Genesis.
We examine our Semantic Genesis with all the publicly-available pre-trained models, by either self-supervision or fully supervision, on the six distinct target tasks, covering both classification and segmentation in various medical modalities (i.e., CT, MRI, and X-ray).
[Conclu] A key contribution of ours is designing a self-supervised learning framework that not only allows deep models to learn common visual representation from image data directly, but also leverages semantics-enriched representation from the consistent and recurrent anatomical patterns, one of a broad set of unique properties that medical imaging has to offer.
Methods: Semantic Genesis
They represent a significant advancement from Models Genesis [25] by introducing two novel components: self-discovery and self-classification of the anatomy underneath medical images (detailed in Sec. 2).
Specifically, our unique self-classification branch, with a small computational overhead, compels the model to learn semantics from consistent and recurring anatomical patterns discovered during the selfdiscovery phase, while Models Genesis learns representation from random subvolumes with no semantics as no semantics can be discovered from random sub-volumes.
Results:
Our extensive experiments demonstrate that Semantic Genesis significantly exceeds all of its 3D counterparts as well as the de facto ImageNet-based transfer learning in 2D.
This performance is attributed to our novel self-supervised learning framework, encouraging deep models to learn compelling semantic representation from abundant anatomical patterns resulting from consistent anatomies embedded in medical images.
[Conclu] Our extensive results demonstrate that Semantic Genesis is superior to publicly available 3D models pre-trained by either self-supervision or even full supervision, as well as ImageNet-based transfer learning in 2D. We attribute this outstanding results to the compelling deep semantics learned from abundant anatomical patterns resulted form consistent anatomies naturally embedded in medical images.
Semantic Genesis is conceptually simple: an encoder-decoder structure with skip connections in between and a classification head at the end of the encoder.
The objective is to learn different sets of semantics-enriched representations from multiple perspectives.
Proposed framework consists of three important components:
Self-discovery of anatomical patterns from similar patients;
Self-classification of the patterns;
Self-restoration of the transformed patterns.
Specifically, once the self-discovered anatomical pattern set is built, we jointly train the classification and restoration branches together in the model.
Fig. 1. Our self-supervised learning framework consists of (a) self-discovery, (b) selfclassification, and (c) self-restoration of anatomical patterns, resulting in semanticsenriched pre-trained models—Semantic Genesis—an encoder-decoder structure with skip connections in between and a classification head at the end of the encoder. Given a random reference patient, we find similar patients based on deep latent features, crop anatomical patterns from random yet fixed coordinates, and assign pseudo labels to the crops according to their coordinates. For simplicity and clarity, we illustrate our idea with four coordinates in X-ray images as an example. The input to the model is a transformed anatomical pattern crop, and the model is trained to classify the pseudo label and to recover the original crop. Thereby, the model aims to acquire semanticsenriched representation, producing more powerful application-specific target models.
References:
https://medium.com/@dptmn200/unsupervised-deep-embedding-for-clustering-analysis-a-summary-f6e5f2dce94f
Page updated
Google Sites
Report abuse