{Hybrid Transformer, Weakly Semi-supervise Learning}
Personally, The most important thing to study from this paper is to utilize image-labeled points for Weakly Semi-Supervised Object Detection and take the benefit from the positional encoder of DERT
0. Motivation, Objective, and Related Works:
Motivation:
Very costly to annotate a large amount of detection data.
Find that existing detectors have difficulty in fully exploiting the power of the annotated points.
Objective:
We propose a novel point annotated setting for the weakly semi-supervised object detection task, in which the dataset comprises small fully annotated images and large weakly annotated images by points.
Annotate each instance in the image by one point (as shown in Figure 1d) instead of image-level annotation, for two main reasons. Firstly, compared with image-level annotation, points bring much richer information, not only the category of the object but also the strong prior of object location. Secondly, there is no strict requirement on point annotations, such as center points of objects.
Propose a new simple network architecture: Points as Queries
Related Works:
Supervised Object Detection:
With the large-scale fully annotated detection data, existing modern detectors have obtained great improvements in the object detection task.
Two-stage detectors and one-stage detectors predicts object proposals firstly and refines these proposals finally.
One-stage detectors directly output the classification and location of each object without refinement. Though achieving great success, these detectors are trained with a large amount of fully-annotated data, which is costly to annotate. Thus, there are many works proposed to reduce the annotation cost.
Semi-/ Weakly-/ Weakly Semi- Supervised Object Detection:
Semi-supervised object detection methods: learn detectors with a small amount of box-level labeled images and large unlabeled images. (Jeong et al. [10] employ consistency constraints for object detection to exploit unlabeled data).
Weakly supervised object detection methods utilize large data with weak annotations, such as image labels. (Bilen et al. [2] learn an object detector under image-level supervision by combining region classification and selection).
Weakly semi-supervised object detection methods (WSSOD): Pursuing the performance of supervised detection and keeping the low cost of annotation. (Use small box-level labeled images as well as large weakly labeled images to learn detectors).
Point-based Semi-Supervised Segmentation:
Has been employed by semantic segmentation.
Bearman et al. [1] incorporate point supervision along with objectness prior to boost segmentation performance and alleviate annotation burden.
Qian et al. [19] leverage semantic relationships among several labeled points to address the semantic scene parsing task.
DETR:
By virtue of Transformer [30], DETR takes an image as input and directly outputs a fixed set of box predictions. For the point-based detection task, DETR has a beneficial characteristic: a single-level feature map, avoiding the multi-level selection problem
Figure 1. Different types of object detection settings to reduce the cost of data annotation
Contribution:
We propose a potential and novel setting for the weakly semi-supervised object detection task, which comprises small fully annotated images and large weakly annotated images by points.
Analyze the drawbacks of existing modern object detectors and propose Point DETR. The proposed detector takes object points as input, transforms these points into object queries, and predicts object box precisely for these queries. (different from DETR, (1) we encode position and category of annotated points into object queries with the point encoder, which easily establishes one-to-one correspondences between points and object queries, being fit for box prediction based on points; (2) do box predictions as offsets w.r.t. point position rather than make box predictions directly like DETR.)
Extensive experiments on COCO dataset [16] are conducted to demonstrate the effectiveness of our proposed detector. Our detector outperforms most modern detectors in various data settings. We also do quantity and quality experiments to show our detector solves the problems suffered by most modern detectors.
1. Methods:
Overall Framework:
Adapt self-training as our default training pipeline:
Train a teacher model on available labeled images.
Generate pseudo-labels of weakly point annotated images using the trained teacher model.
Train a student model with fully labeled images and pseudo-labeled images.
For most self-training-based detection methods, hyper-parameters are selected carefully since they must keep true object boxes and screen out false ones as much as possible.
Instead, we can directly predict the corresponding object box for each point annotation without duplicate object boxes.
Although choosing hyper-parameters is no longer an obstacle to performance, predicting object boxes from point-level annotations with existing detectors remains a problem.
