[OTA]Optimal Transport Assignment for Object Detection(CVPR. 2021)

2023-11-29 08:03:22

1. Motivation

DeTR [3] examines the idea of global optimal matching. But the Hungarian algo- rithm they adopted can only work in a one-to-one assign- ment manner.
One-to-Many 的方法。

So far, for the CNN based detectors in one-to-many scenarios, a global optimal assigning strategy remains uncharted.
Label Assignment

To train the detector, defining cls and reg targets for each anchor is a necessary procedure, which is called label assignment in object detection.
Such static strategies ignore a fact that for objects with different sizes, shapes or occlusion condition, the appropriate posi- tive/negative (pos/neg) division boundaries may vary.

文中认为对于ambiguous anchors的制定是非常重要的。

Hence the assignment for ambiguous anchors is non-trivial and requires further information beyond the local view.

要将独立的最优分配转化为全局的最优分配。

Thus a better assigning strategy should get rid of the convention of pursuing optimal assignment for each gt independently and turn to the ideology of global optimum, in other words, finding the global high confidence assignment for all gts in an image.

2. Contribution

相比于DETR的one-to–one Label Assignment，本文认为One-to-Many的Lbael Assignment同样可以对训练有帮助，也可以将制定带有global view的labels。

OT将anchor看做demander，将gt看做supplier。每一个gt供应positive label的数量看做为“每一个gt需要多少个positive anchor来完成训练过程，更好的收敛“。

OTA分别要求anchor与gt以及anchor与background pair-wise的loss，其中anchor与gt pair的transportation cost是cls和reg的loss，而anchor与background的pair-wise loss 只需要计算cls loss就好。

To achieve the global optimal assigning result under the one-to-many situation, we propose to formulate label as-signment as an Optimal Transport (OT) problem – a special form of Linear Programming (LP) in Optimization Theory.
we define each gt as a supplier who supplies a certain number of labels, and define each anchor as a de- mander who needs one unit label.
In this context, the number of positive labels each gt supplies can be interpreted as “how many positive anchors that gt needs for better convergence during the training process”.
The unit transportation cost between each anchor-gt pair is defined as the weighted summation of their pair-wise cls and reg losses.
The cost between background and a certain anchor is defined as their pair-wise classification loss only.
OTA also achieves the SOTA performance among one-stage detectors on a crowded pedestrian detection dataset named CrowdHu- man [35], showing OTA’s generalization ability on different detection benchmarks

3.Method

3.1. Optimal Transport

Transporting cost for each unit of good from supplier i to demander j is denoted by c i j c_{ij} cij
We thus address this issue by a fast iterative solution, named Sinkhorn-Knopp

3.2 OT for Label Assignment

m gt targets and n anchors. 根据one-to-many的关系，一个supplier有多个unit（一个unit对应一个demander），一个demander(anchor)值对应一个supplier(gt)。

we view each gt as a supplier who holds k units of positive labels ( s i = k s_i=k si=k, i = 1, 2, …, m)。
each anchor as a demander who needs one unit of label(i.e. d j = 1 d_j = 1 dj=1, j= 1,2,…, n)。

c f g c^{fg} cfg前景cost的公式如下所示：（one unit）

其中，Lcls和Lreg分别是cross entropy loss 和 IoU Loss（也可以被其他常用损失函数取代）。α是平衡参数。

对于 c b g c^{bg} cbg背景cost的公式如下所示：

negative labels的数量为 n − m × k n-m\times k n−m×k。 m × k m \times k m×k表示gt共有的所有units，而n表示anchor的个数，由于每一个unit就对应一个anchor（demander）。因此剩余的数量就要分配为negative labels。

c b g ∈ R 1 × n c^{bg}\in R^{1\times n} cbg∈R1×n, c f g ∈ R m × n c^{fg} \in R^{m \times n} cfg∈Rm×n。cat起来可以得到最后的cost matrix c ∈ R ( m + 1 ) × n c \in R^{(m+1) \times n} c∈R(m+1)×n

supply vector s的公式如下所示：

因此可以得到optimal transportation plan π ∗ ∈ R ( m + 1 ) × n \pi^* \in R^{(m+1)\times n} π∗∈R(m+1)×n。

接下来使用线程的Sinkhorn-Knopp Iteration方法，为每一个anchor制定一个gt，这个gt是传输了最大数量的label。

After getting π∗, one can decode the correspond- ing label assigning solution by assigning each anchor to the supplier who transports the largest amount of labels to them.

OTA在训练中有一定的增加，测试无损耗。

OTA only increases the total training time by less than 20% and is totally cost-free in testing phase.

3.3 Advanced Designs

3.3.1 Center Prior

加入center prior有利于在前期训练稳定，并且提升网络性能。

* However, for general detection datasets like COCO, we find the Center Prior still benefit the training of OTA.
* Hence, we impose a Center Prior to the cost matrix. 
* For each gt, we select $r^2$ closest anchors from each FPN level according to the center distance between anchors and gts.

As for anchors not in the r 2 r^2 r2 closest list, their corresponding entries in the cost matrix c will be subject to an additional constant cost to reduce the possibility they are assigned as positive samples during the training stage.

3.3.2 Dynamic k Estimation

对于每个gt对应k个anchor的选取，作者采用简单基于IOU的方法来制定。

we propose a simple but effective method to roughly estimate the appropriate number of pos- itive anchors for each gt based on the IoU values between predicted bounding boxes and gts.

对于每个gt，挑选top q高的iou的 predicted-box，并将iou值全部加起来，来代表每个gt估算的positive anchors的数量，也就是公式中的k。作者将这个方法，命名为Dynamic k Estimation。

Dynamic k Estimation： Specifically, for each gt, we select the top q predictions according to IoU values. These IoU values are summed up to represent this gt’s estimated number of positive anchors.
完整的算法如下：

4. Experiments

4.1 Ablation Studies and Analysis

4.1.1 Effects of Individual Components.

4.1.2 Effects of r

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gp9Uj7wh-1626247383540)(https://i.loli.net/2021/07/14/3cshG4HlLpMd8gv.png)]