[SSA] Semantic Segment Anything

Motivation:

• Semantic Segment Anything (SSA) project enhances the Segment Anything dataset (SA-1B) with a dense category annotation engine. 

Objectives:

• SSA is an automated annotation engine that serves as the initial semantic labeling for the SA-1B dataset. 

• While human review and refinement may be required for more accurate labeling. Thanks to the combined architecture of close-set segmentation and open-vocabulary segmentation, SSA produces satisfactory labeling for most samples and has the capability to provide more detailed annotations using the image caption method. 

• This tool fills the gap in SA-1B's limited fine-grained semantic labeling, while also significantly reducing the need for manual annotation and associated costs. 

• It has the potential to serve as a foundation for training large-scale visual perception models and more fine-grained CLIP models.

Introduction:

Related Works:

Contribution:

Method 1:

• The SSA engine consists of three components: 

  1. Close-set semantic segmentor (green): Two close-set semantic segmentation models trained on COCO and ADE20K datasets respectively are used to segment the image and obtain rough category information. The predicted categories only include simple and basic categories to ensure that each mask receives a relevant label.

  2. Open-vocabulary classifier (blue): An image captioning model is utilized to describe the cropped image patch corresponding to each mask. Nouns or phrases are then extracted as candidate open-vocabulary categories. This process provides more diverse category labels.

  3. Final decision module (orange): The SSA engine uses a Class proposal filter (i.e. a CLIP) to filter out the top-k most reasonable predictions from the mixed class list. Finally, the Open-vocabulary Segmentor predicts the most suitable category within the mask region based on the top-k classes and image patch.

Method 2:

Dataset:

Metrics:

Experimental Results:

Ablations:

References:

1. • n2  n0

   • θ