03_Segmentation

Image segmentation, which aims to recognize an image at the pixel level, is a fundamental yet challenging problem in computer vision. Image segmentation can be formulated as different tasks:

Semantic segmentation: aims to assign each pixel a category label (e.g., for uncountable stuff classes).
Instance segmentation: aims to assign each object a pixel-wise instance mask and a category label (e.g., for countable things).
Panoptic segmentation: unifies semantic and instance segmentation, which aims to assign each pixel a semantic label and a unique instance identity (for both stuff and things).

"Improving Image Segmentation with Boundary Patch Refinement"

Overview

Dataset

Metrics

Precision and Recall (Sensitivity)
Accuracy/ Rand index.
Dice Coefficient.
Jaccard index (IoU).

Papers

Survey

2D Segmentation

"Image segmentation using deep learning: A survey", 2020.
R Sharma, M Saqib, CT Lin, M Blumenstein, "A Survey on Object Instance Segmentation", SN Computer Science, 2022
W Gu, S Bai, L Kong, "A review on 2D instance segmentation based on deep neural networks", Image and Vision Computing, 2022
I Ulku, E Akagündüz, "A survey on deep learning-based architectures for semantic segmentation on 2d images", Applied Artificial Intelligence, 2022

3D Segmentation

Y. He, H. Yu, X. Liu, Z. Yang, W. Sun, Y. Wang, Q. Fu, Y. Zou, and A. Mian, "Deep Learning based 3D Segmentation: A Survey", arXiv preprint arXiv:2103.05423, 2021.

Medical Imaging

- D. Shen, G. Wu, and H.-I. Suk, “Deep learning in medical image analysis,” Annual review of biomedical engineering, vol. 19, pp. 221–248, 2017.
- N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. N. Chiang, Z. Wu, and X. Ding, “Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation,” Medical Image Analysis, vol. 63, p. 101693, 2020.
- R. Wang, T. Lei, R. Cui, B. Zhang, H. Meng, and A. K. Nandi, “Medical image segmentation using deep learning: A survey,” IET Image Processing, Jan 2022.
- S Niyas, SJ Pawan, MA Kumar, J Rajan, "Medical Image Segmentation with 3D Convolutional Neural Networks: A Survey", Neurocomputing, 2022.
- "Medical image segmentation model based on triple gate MultiLayer perceptron"

Boundary Detection

- "A review on lung boundary detection in chest X-rays"
- "Yet Another Survey on Image Segmentation: Region and Boundary Information Integration", ECCV, 2020.

Semantic Segmentation

"Active Boundary Loss for Semantic Segmentation"
"G2L: A Global to Local Alignment Method for Unsupervised Domain Adaptive Semantic Segmentation"
[FCNs] "Fully Convolutional Networks for Semantic Segmentation", 2015.
[DIC] L. Zhou, and W. Wei, "DIC: Deep Image Clustering for Unsupervised Image Segmentation, IEEE Access, 2020. (segmentation + clustering)
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation(SegNet)
U-Net: Convolutional Networks for Biomedical Image Segmentation(UNet)
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs(Deeplab v1)
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution,and Fully Connected CRFs(Deeplab v2)
Understanding Convolution for Semantic Segmentation(DUC)
Pyramid Scene Parsing Network(PSPNet)
Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network(GCN)
Rethinking Atrous Convolution for Semantic Image Segmentation(Deeplab v3)
DenseASPP for Semantic Segmentation in Street Scenes（DenseASPP）
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation(Deeplab v3plus）
Context Encoding for Semantic Segmentation(EncNet)
Learning a Discriminative Feature Network for Semantic Segmentation(DFN)
Smoothed Dilated Convolutions for Improved Dense Prediction(SDC)
Pyramid Attention Network for Semantic Segmentation(PAN)
Exploring Context with Deep Structured models for Semantic Segmentation(FeatMap-Net)
ExFuse: Enhancing Feature Fusion for Semantic Segmentation(ExFuse)
Dilated Residual Networks(DRN)
Dual Attention Network for Scene Segmentation(DANet)
OCNet:Object Context Network for Scene Parsing(OCNet)
RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation(RefineNet)
Dense Relation Network: Learning Consistent And Context-Aware Prepresentation For Semantic Image Segmentation(DRN)
CCNet: Criss-Cross Attention for Semantic Segmentation(CCNet)
Unified Perceptual Parsing for Scene Understanding(UPerNet)
Tree-structured Kronecker Convolutional Networks for Semantic Segmentation(TKNet)
NeuroIoU: Learning a Surrogate Loss for Semantic Segmentation(NeuroIoU)
Decoders Matter for Semantic Segmentation:Data-Dependent Decoding Enables Flexible Feature Aggregation
GFF: Gated Fully Fusion for Semantic Segmentation（GFF）
Learning Fully Dense Neural Networks for Image Semantic Segmentation（FDNet）
ZigZagNet: Fusing Top-Down and Bottom-Up Context for Object Segmentation(ZigZagNet)
Adaptive Pyramid Context Network for Semantic Segmentation(APCNet)
Dense Decoder Shortcut Connections for Single-Pass Semantic Segmentation
ACFNet: Attentional Class Feature Network for Semantic Segmentation(ACFNet)
Miss Detection vs. False Alarm: Adversarial Learning for Small Object Segmentation in Infrared Images
Dual Graph Convolutional Network for Semantic Segmentation
Global Aggregation then Local Distribution in Fully Convolutional Networks
Dynamic Multi-scale Filters for Semantic Segmentation
Unifying Training and Inference for Panoptic Segmentation
Semantic Flow for Fast and Accurate Scene Parsing
AlignSeg: Feature-Aligned Segmentation Networks
Cars Can’t Fly up in the Sky: Improving Urban-Scene Segmentation via Height-driven Attention Networks
Context Prior for Scene Segmentation

Increase Feature Maps Resolution (Semantic)

"Rethinking atrous convolution for semantic image segmentation", 2017.
"Encoder-decoder with atrous separable convolution for semantic image segmentation", 2018.

Enrich Context Information (Semantic)

"Dual attention network for scene segmentation", 2019.
"Objectcontextual representations for semantic segmentation", 2019.
"Co-occurrent features in semantic segmentation", 2020.
"Pyramid scene parsing network", 2017.

Use Encoder-Decoder Architecture (Semantic)

"Encoder-decoder with atrous separable convolution for semantic image segmentation", 2018.
"Panoptic feature pyramid networks", 2019.
"V-net: Fully convolutional neural networks for volumetric medical image segmentation", 2016.
"Unet: Convolutional networks for biomedical image segmentation", 2015.

Attention-based

[VAN] "Visual Attention Network". (Large Kernel Attention - LKA + Channel Attention)
[SegNeXt] "SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation" (Multi Scale Conv Attention - MSCA)

Transformer

"A Unified Efficient Pyramid Transformer for Semantic Segmentation", Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 2667-2677, 2021.

Application

T. Wan, L. Zhao, H. Feng, D. Li, C. Tong, and Z. Qin, "Robust nuclei segmentation in histopathology using ASPPU-Net and boundary refinement", Neurocomputing, 2020.
Flood Segmentation on Sentinel-1 SAR Imagery with Semi-Supervised Learning_2021 [Paper]

Medical Image

[nnU-Net] F Isensee, J Petersen, A Klein, D Zimmerer, PF Jaeger, Simon Kohl, Jakob Wasserthal, Gregor Koehler, Tobias Norajitra, Sebastian Wirkert, Klaus H. Maier-Hein, "nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation", arXiv preprint arXiv:1809.10486, 2018.

Instance Segmentation

"Mask R-CNN"
"Mask Scoring R-CNN", 2019
[PANet] "Path Aggregation Network for Instance Segmentation", 2018.

One-stage Instance Segmentation

[YOLACT] "YOLACT Real-time Instance Segmentation", 2019.
"Blendmask: Top-down meets bottom-up for instance segmentation", 2020.
"Tensormask: A foundation for dense object segmentation", 2019.
"Centermask: Real-time anchor-free instance segmentation", 2020.
[MEInst]
[CondInst] "Conditional convolutions for instance segmentation", 2020.
[SOLO] "Solo: Segmenting objects by locations", 2020.
[SOLOv2] "Solov2: Dynamic and fast instance segmentation", 2020.
"Embedmask: Embedding coupling for one-stage instance segmentation", 2019.
"Mask encoding for single shot instance segmentation", 2020.

Other Instance Segmentation

[Sparse R-CNN]
[DANCE] Z Liu, JH Liew, X Chen, J Feng, "DANCE: A Deep Attentive Contour Model for Efficient Instance Segmentation", Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 345-354, 2021.
Fully Convolutional Instance-aware Semantic Segmentation(FCIS)
Instance-aware Semantic Segmentation via Multi-task Network Cascades(MNC)
RetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free(RetinaMask)
Parsing R-CNN for Instance-Level Human Analysis(Parsing R-CNN)
BlitzNet: A Real-Time Deep Network for Scene Understanding(BlitzNet)
Hybrid Task Cascade for Instance Segmentation(HTC)
Triply Supervised Decoder Networks for Joint Detection and Segmentation(TripleNet)
ZigZagNet: Fusing Top-Down and Bottom-Up Context for Object Segmentation(ZigZagNet)
Bounding Box Embedding for Single Shot Person Instance Segmentation
Shape-aware Feature Extraction for Instance Segmentation
Real-Time Panoptic Segmentation from Dense Detections
EmbedMask: Embedding Coupling for One-stage Instance Segmentation
PolyTransform: Deep Polygon Transformer for Instance Segmentation
SOLO: Segmenting Objects by Locations_ECCV_2020 [Paper] [Personal Summary]
RDSNet: A New Deep Architecture for Reciprocal Object Detection and Instance Segmentation
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid
YOLACT++：Better Real-time Instance Segmentation
SAIS: Single-stage Anchor-free Instance Segmentation
"PolarMask: Single Shot Instance Segmentation with Polar Representation", 2020
BANet: Bidirectional Aggregation Network with Occlusion Handling for Panoptic Segmentation
[CenterMask] Y. Lee, and J. Park, "CenterMask: Real-Time Anchor-Free Instance Segmentation", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13906-13915, 2020.

Application

YolactEdge: Real-time Instance Segmentation on the Edge [Link] [Code]
L-Seg: An end-to-end unified framework for multi-lesion segmentation of fundus images

Medical Image

CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation (Different Attention: Channel, Spatial, Scale)

Transformer - Pure

[Hrformer] Y. Yuan, R. Fu, L. Huang, W. Lin, C. Zhang, X. Chen, and J. Wang, “Hrformer: High-resolution vision transformer for dense predict,” Advances in Neural Information Processing Systems, vol. 34, pp. 7281–7293, 2021.
"Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation"

Transformer - Hybrid

[DETR]
[ISTR] J. Hu, L. Cao, Y. Lu, S. C. Zhang, Y. Wang, K. Li, F. Huang, L. Shao, and R. Ji, "ISTR: End-to-End Instance Segmentation with Transformers", arXiv preprint arXiv arXiv:2105.00637v2, 2021
[SOIL] X Yu, D Shi, X Wei, Y Ren, T Ye, W Tan, "SOIT: Segmenting Objects with Instance-Aware Transformers", Proceedings of the AAAI Conference on Artificial Intelligence, 36(3), 3188-3196, 2022.

Others

FaPN: Feature-aligned Pyramid Network for Dense Image Prediction_2021 [Paper]
Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs_2021 [Paper]
[SegCaps] Capsules for Object Segmentation [Paper] [Code]
"Two-Stage Segmentation Framework Based on Distance Transformation"

Real-time

"Deep Snake for Real-Time Instance Segmentation"

Panoptic Segmentation

Real-time/ Light-weight Segmentation

"Explicit shape encoding for real-time instance segmentation", 2019.
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation(ENet)
ICNet for Real-Time Semantic Segmentation(ICNet)
BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation(BiSeNet)
LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation(LinkNet)
Rtseg: Real-Time Semantic Segmentation Comparative Study
Shuffleseg: Real-Time Semantic Segmentation Network(Shuffleseg)
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation(ESPNet)
Light-Weight RefineNet for Real-Time Semantic Segmentation(Light-Weight RefineNet)
LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation(LinkNet)
D-LinkNet: LinkNet with Pretrained Encoder and Dilated Convolution for High Resolution Satellite Imagery Road Extraction(D-LinkNet)
CGNet: A Light-weight Context Guided Network for Semantic Segmentation(CGNet)
Efficient ConvNet for Real-time Semantic Segmentation
A Comparative Study of Real-time Semantic Segmentation for Autonomous Driving
ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time(ContextNet)
ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network(ESPNetV2)
ShelfNet for Real-time semantic segmentation(ShelfNet)
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation(ERFNet)
Concentrated-Comprehensive Convolutions for lightweight semantic segmentation(CCCNet)
DSNet for Real-Time Driving Scene Semantic Segmentation(DSNet)
Efficient Dense Modules of Asymmetric Convolution for Real-Time Semantic Segmentation(EDANet)
Fast-SCNN: Fast Semantic Segmentation Network(Fast-SCNN)
Guided Upsampling Network for Real-Time Semantic Segmentation(GUN)
In Defense of Pre-trained ImageNet Architecturesfor Real-time Semantic Segmentation of Road-driving Images(SwiftNetRN)
Residual Pyramid Learning for Single-Shot Semantic Segmentation(RPNet)
DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation(DFANet)
DSNet: An Efficient CNN for Road Scene Segmentation(DSNet)
Spatial Sampling Network for Fast Scene Understanding
RGPNET: A REAL-TIME GENERAL PURPOSE SEMANTIC SEGMENTATION
LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation
FASTERSEG: SEARCHING FOR FASTER REAL-TIME SEMANTIC SEGMENTATION
Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
Customizable Architecture Search for Semantic Segmentation
Semantic Flow for Fast and Accurate Scene Parsing
BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation
ASNet: Aggregated Scale Transformations for Real-Time Semantic Segmentation

Medical Image Segmentation

CNN-based

- Q. Liu, X. Tang, D. Guo, Y. Qin, P. Jia, Y. Zhan, X. Zhou, and D. Wu, “Multi-class gradient harmonized dice loss with application to knee mr image segmentation,” in MICCAI, pp. 86– 94, Springer, 2019.

Transformer-based

- [Transunet] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, “Transunet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306, 2021.
- [Transfuse] Y. Zhang, H. Liu, and Q. Hu, “TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation”, in MICCAI, pp. 14–24, Springer, 2021.
- [Utnet] Y. Gao, M. Zhou, and D. N. Metaxas, “Utnet: a hybrid transformer architecture for medical image segmentation,” in MICCAI, pp. 61–71, Springer, 2021.
- [Segformer] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in Neural Information Processing Systems, vol. 34, pp. 12077–12090, 2021.
- [Swin-unet] H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, and M. Wang, “Swin-unet: Unet-like pure transformer for medical image segmentation,” arXiv preprint arXiv:2105.05537, 2021.
- [UNETR] A Hatamizadeh, Y. Tang, V. Nath, D.. Yang, A Myronenko, B. Landman, H. Roth, and D. Xu, "UNETR: Transformers for 3D Medical Image Segmentation", IEEE Winter Conference on Applications of Computer Vision (WACV), 2022.
- [Swin UNETR] "Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images"
- [VT-UNet]
- "ISTR: End-to-End Instance Segmentation with Transformers"

Attention-based

- [CBAM-TransUNet)] X. Chen, and L. Yang, "Brain tumor segmentation based on CBAM-TransUNet", Proceedings of the 1st ACM Workshop on Mobile and Wireless Sensing for Smart Healthcare (MWSSH), 2022.

Semi-supervised

- X Luo, J Chen, T Song, G Wang, "Semi-supervised Medical Image Segmentation through Dual-task Consistency", The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), 2021.

Mask-Refinement

- H Tang, X Liu, S Sun, X Yan, X Xie, "Recurrent Mask Refinement for Few-Shot Medical Image Segmentation", Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3918-3928, 2021.

Boundary Refinement

Instance

- [BPR] C. Tang, H. Chen, X. Li, J. Li, Z. Zhang, and X. Hu, "Boundary Patch Refinement for Instance Segmentation", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13926-13935, 2021. [Code]
- [BPR] X. Hu, C. Tang, H. Chen, X. Li, J. Li, and Z. Zhang, "Improving Image Segmentation with Boundary Patch Refinement", International Journal of Computer Vision, 2022. (Mask RCNN + Patch Extraction + Boundary Refinement)
- [PointRend] "Pointrend: Image segmentation as rendering", 2020
- [Focal-BG] "Focal boundary guided salient object detection", 2019.
- [PolyTransform] "Polytransform: Deep polygon transformer for instance segmentation", 2020.
- [Deepstrip] "Deepstrip: High-resolution boundary refinement", 2020.
- [BMask-RCNN[ T. Cheng, X. Wang, L. Huang, and W. Liu, "Boundary-Preserving Mask R-CNN", European conference on computer vision, 2020. (Mask RCNN + Boundary Loss)

Semantic

"Efficient inference in fully connected crfs with gaussian edge potentials", 2011.
"Iterative instance segmentation", 2016.
[SegFix] "Segfix: Model-agnostic boundary refinement for segmentation", 2020.
[HMEDN] "High-resolution encoder-decoder networks for low-contrast medical image segmentation", 2019.
[Gated-SCNN] "Gated-scnn: Gated shape cnns for semantic segmentation", 2019.
[DecoupleSegNets] "Improving semantic segmentation via decoupled body and edge supervision", 2020.
"Multitask Semantic Boundary Awareness Network for Remote Sensing Image Segmentation"
"Remote Sensing Image Semantic Segmentation Based on Edge Information Guidance"
"RANet: Region Attention Network for Semantic Segmentation"
"Joint Semantic Segmentation and Boundary Detection Using Iterative Pyramid Contexts", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13666-13675, 2020.
"Label Refinement Network for Coarse-to-Fine Semantic Segmentation"

Others

"Strategies for image segmentation combining region and boundary information"
"End-to-End Boundary Aware Networks for Medical Image Segmentation"
"Boundary-Aware Segmentation Network for Mobile and Web Applications"
"Enhanced Boundary Learning for Glass-Like Object Segmentation"
"Look at Boundary: A Boundary-Aware Face Alignment Algorithm"
"Boundary loss for highly unbalanced segmentation"

Transformer

"DPBET: A dual-path lung nodules segmentation model based on boundary enhancement and hybrid transformer"

3D Segmentation

Y. Chen, "Deep Learning based 3D Image Segmentation Methods and Applications", 2019. (Thesis)

References

https://github.com/michaelowenliu/awesome-semantic-segmentation

Page updated

Google Sites

Report abuse

03_Segmentation

Overview

Dataset

Metrics

Papers

Survey

2D Segmentation

3D Segmentation

Medical Imaging

Boundary Detection

Semantic Segmentation

Increase Feature Maps Resolution (Semantic)

Enrich Context Information (Semantic)

Attention-based

Transformer

Application

Medical Image

Instance Segmentation

One-stage Instance Segmentation

Other Instance Segmentation

Application

Medical Image

Transformer - Pure

Transformer - Hybrid

Others

Real-time

Panoptic Segmentation

Real-time/ Light-weight Segmentation

Medical Image Segmentation

CNN-based

Transformer-based

Attention-based

Semi-supervised

Mask-Refinement

Boundary Refinement

Instance

Semantic

Others

Transformer

3D Segmentation

References

About Me: