Motivation

Objectives

• Different from SORT which uses only the IOU-based distance for object association, DeepSORT combines it with a visual appearance distance obtained from the ReID model. In particular, when a detection i is compared with an existing tracklet j, their visual appearance features, i.e., the embeddings, are compared to obtain their cosine distance. A weighted sum between the IOU distance and the cosine distance is then computed and used as the cost for object association.

• Another difference between SORT and DeepSORT is that DeepSORT performs a cascade matching, in which tracklets with a smaller age are compared and associated with new detections before those with a larger age. The term age in this context refers to the number of consecutive frames that a tracklet has not been matched with any new detection. If it is matched in the current frame, the age is reset to zero; otherwise, it is increased by one. The method, therefore, gives priority to objects that have been seen more recently.

• After these age-based cascade matching, the original SORT association, which is based solely on the IOU distance, is applied as the final step for the unmatched tracklets and detections. This final step is done to take into account the case of sudden changes in visual appearance, which might occur from time to time.

Problem of Sort

1. Linear Assumption: Sort uses Linear Kalman Filter <= not suitable.

2. ID Switches: the relationship btw Detect and Track is IoU => shape

   • The repeatation on ID Switches while being obscured or overlapped.

Ideas

• The factors the improve the performance:

  1. Data Association: Database links between detected objects and saved tracks

  2. Track Life Cycle Management: Manage when tracks begin, stop and remove.

• In Sort: Data Association => Hungarian Algorithm based on IoU (weakness). Link is also associated with:

  1. The distance between detection and track (the correlation in vector space)

  2. The cosine distance between 2 representative vectors extracted by detection and track. (same if belonging to 1 object and different if belong to 2 different objects)

Training with an extracted representative

• Architecture: Wide Residual Network

• Dataset: large scale re-id dataset (Market 1501, MARs...)

• Paper: Deep Cosine Metric Learning for Person Re-Identification (2018) [Paper]

Wide Residual Network (WRN) 

• Deep networks are costly for training and inferencing.

• Use the shallow network => WRN with 16 layers.

Cosine softmax classifer:

• Parameterize the standard softmax classifier. (Above is standard and below is cosine)

• 3 Loss:

  1. Softmax Cross Entropy

  2. Magnet Loss

  3. Triplet Loss

• Cross-Entropy returns good result

New measurements

DeepSort

• Each track is composed of 8 components:

  1. u,v: the center of BBox.

  2. gamma: the frame ratio.

  3. h: the height of BBox.

  4. other: the corresponding velocities.

• Process:

  1. Detection: Faster Region CNN (VGG16 backbone)

  2. New track situations prediction: Kalman filter

-  Tentative (new)

-  Confirmed (3 frames and next 30 frames)

-  Deleted

• 3. Matching Cascade: Linking the detected objects with the confirmed tracks based on the distance measurement.

  4. Hungarian Algorithm: Existing tracks are assigned by IoU for the second linking.

  5. Process and Classify detections and tracks

  6. Kalman Filter: Correct the value of tracks from linked detections and create new tracks.