Virtual Class Enhanced Discriminative Embedding Learning

11/30/2018 ∙ by Binghui Chen, et al. ∙ Beijing Didi Infinity Technology and Development Co., Ltd. 0

Recently, learning discriminative features to improve the recognition performances gradually becomes the primary goal of deep learning, and numerous remarkable works have emerged. In this paper, we propose a novel yet extremely simple method Virtual Softmax to enhance the discriminative property of learned features by injecting a dynamic virtual negative class into the original softmax. Injecting virtual class aims to enlarge inter-class margin and compress intra-class distribution by strengthening the decision boundary constraint. Although it seems weird to optimize with this additional virtual class, we show that our method derives from an intuitive and clear motivation, and it indeed encourages the features to be more compact and separable. This paper empirically and experimentally demonstrates the superiority of Virtual Softmax, improving the performances on a variety of object classification and face verification tasks.



There are no comments yet.


page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In the community of deep learning, the Softmax layer is widely adopted as a supervisor at the top of the model due to its simplicity and differentiability, such as in object classification

he2016deep ; hu2017squeeze ; huang2016densely ; zagoruyko2016wide

and face recognition

sun2014deep ; sun2015deeply ; schroff2015facenet ; wen2016discriminative , etc. While, many research works take it for granted and omit the fact that the learned features by Softmax are only separable not discriminative. Thus, the performances of deep models in many recognition tasks are limited. Moreover, there are a few research works concentrating on learning discriminative features through refining this commonly used softmax layer, e.g. L-Softmax liu2016large and A-Softmaxliu2017sphereface . However, they require an annealing-like training procedure which is controlled by human, and thus are difficult to transfer to other new tasks. To this end, we intend to propose an automatic counterpart which dedicates to learning discriminative features.

Figure 1: Illustration of angularly distributed features on 2-D space. (a) shows features learned by the original 4-way softmax. (b) shows features learned by 8-way softmax, where there are 4 additionally hand-injected negative classes.

denotes the class anchor vector.

In standard softmax, the input pattern (with label

) is classified by evaluating the inner product between its feature vector

and the class anchor vector (if not specified, the basis is removed, which does not affect the final performances of DCNN verified in liu2016large ; liu2017sphereface ). Since the inner product can be rewritten into , where is the angle between vector and , and the linear softmax classifier is mainly determined by the angle 111Elucidated in liu2016large . Here for simplicity, assume that all ., thus can be regarded as an angle classifier. Moreover, as shown in wang2017normface , the feature distribution leaned by softmax is ’radial’, like in Fig.1. Thus, learning the angularly discriminative feature is the bedrock of softmax-based classifier. Here, we will first succinctly elucidate the direct motivation of our Virtual Softmax. For a 4-class classification problem, as illustrated in Fig.1.(a)222N-dimensionality space complicates our analysis but has the similar mechanism as 2-D space., one can observe that when optimizing with the original 4 classes, the decision boundary for class overlaps with that for class , i.e. there is no inter-class angular margin. Obviously, this constraint of overlapped boundaries is not enough to ensure intra-class compactness and inter-class separability, and then contributes little to recognition performance improvement. While, in expanded 8-category constraint case as exhibited in Fig.1 .(b), the decision boundaries for class and have a large inter-class angular margin with each other due to the additionally injected class (which doesn’t take part in the final recognition evaluation) and the constrained region of class is much tighter than before (other classes have the same phenomenon), yielding more compact and separable features which are highly beneficial to the original 4-class recognition. Thus, artificially injected virtual classes (e.g. ) can be dedicated to encouraging larger decision margin among original classes (e.g. class ) such that produces angularly discriminative features.

Naturally motivated by this, we propose Virtual Softmax, a novel yet extremely simple technique to improve the recognition performances by training with an extra dynamic virtual class. Injecting this virtual class aims at imposing stronger and continuous constraint on decision boundaries so as to further produces the angularly discriminative features. Concretely, generalizing Softmax to Virtual Softmax gives chances of enlarging inter-class angular margin and tightening intra-class distribution, and allows the performance improvement of recognition via introducing a large angular margin. Different from L-Softmaxliu2016large and A-Softmaxliu2017sphereface , our work aims at extending Softmax to an automatic counterpart, which can be easily performed without or with little human interaction. Fig.2 verifies our technique where the learned features by our Virtual Softmax turn to be much more compact and well separated. And the main contributions of this paper are summarized as follows:

We propose Virtual Softmax to automatically enhance the representation power of learned features by employing a virtual negative class. The learned features are much more compact and separable, illustrated in Fig.2. And to our best knowledge, it is the first work to employ additional virtual class in Softmax to optimize the feature learning and to improve the recognition performance.

The injected virtual class derives from the natural and intuitive motivation, and intends to force a zero limit which is a much stronger constraint than softmax. Moreover, it is dynamic and adaptive, pursuing an automatic training procedure and without incurring much computational complexity and memory consumption.

Extensive experiments have been conducted on several datasets, including MNIST lecun1998mnist , SVHN netzer2011reading , CIFAR10/100 krizhevsky2009learning , CUB200 welinder2010caltech , ImageNet32Chrabaszcz2017A , LFW huang2007labeled and SLLFW deng2017fine . Finally, our Virtual Softmax has achieved competitive and appealing performances, validating its effectiveness.

2 Related Work

Since the training of deep model is dominated and guided by the top objective loss, and this objective function can be equipped with more semantic information, many research works force stronger representation by refining the supervisor at the top of deep model. In sun2014deep

, contrastive loss is employed to enlarge the inter-class Euclidean distance and to reduce the intra-class Euclidean distance. Afterwards, it is generalized to triplet loss, the major idea is to simultaneously maximize inter-class distance and minimize intra-class distance, and it is widely applied in numerous computer vision tasks, such as in face verification

schroff2015facenet , in person re-identification cheng2016person , in fine-grained classification wang2016mining , etc. Moreover, many tuple-based methods are developed and perform better on the corresponding tasks, such as Lifted loss oh2016deep , N-pair loss sohn2016improved ; chen2018almn , etc. However, these tuple-based methods constrain the feature learning with multiple instances, and thus require to elaborately manipulate the tuple mining procedure, which is much expensive in computation and is performance-sensitive. In addition, inspired by linear discriminant analysis, center loss wen2016discriminative

breaks away from the tuple-based idea and shows better performances. However, our Virtual Softmax differs with them in that (1) the model is optimized with only the single Virtual Softmax loss not the joint loss functions (e.g. softmax + contrastive-loss). (2) our method can also be applied to improve the classification performances, while the above methods are not as good in classification tasks. Compared to L-Softmax

liu2016large and A-Softmaxliu2017sphereface , this paper heads from a totally different idea that encourages a large margin among classes via injecting additionally virtual class, and dedicates to proposing an automatic method, i.e. Virtual-Softmax.

Figure 2: Visualization of the learned features optimized by original Softmax vs. Virtual Softmax on MNIST dataset. We provide two cases of 2-D and 3-D visualization, which are shown in the left two columns and the right two columns, respectively. From the visualization, it can be observed that our Virtual Softmax possesses a stronger power of forcing compact and separable features.

3 Intuition and Motivation

In this section, we will give a toy example to introduce the immediate motivation of our Virtual Softmax. Define the i-th input data with its corresponding label , where and is the class number. Define the j-th class anchor vector in softmax classifier , where . The output feature of deep model is defined as .

Figure 3:

Toy example for non-uniform distribution case. If the class number increases, the constrained regions will be more compact than before.

In standard softmax, the optimization objective is to minimize the following cross-entropy loss:


here we omit the basis and it does not affect the performance. Obviously, in original softmax optimization, it is to force , in order to correctly classify the input .

Thus we can obtain the following property of softmax, which is the immediate incentive of our Virtual Softmax.

Property 1.

As the number of class anchor vector increases, the constrained region for each class (e.g. the area of each class in Fig.1,3) becomes more and more compact, i.e. will gradually decreases (e.g. in Fig.1,3).


For simplicity, here we consider the 2-D c-class case (n-D c-class case can be generalized as well but is more complicated.). First, assume and they are evenly distributed in feature space, in another word, every two vectors have the same vectorial angle . As stated above, the softmax is to force in order to correctly classify , i.e. , where denotes the angle between feature vector and class vector . Hence, the decision boundary for class is , i.e. , we define as the angle between and the decision boundary of class , and the solution is (see Fig.1. (a) for a toy example). Considering the distribution symmetry of , the angle range of the constrained region for class is , which is inversely proportional to . Thus, if the class number increases, the constrained region of each class will be more and more compact. Moreover, for the and non-uniform distribution case, the analysis is a little more complicated. Because the length of are different, the feasible angles of class and class are also different (see the decision boundary of the case before injecting new classes in Fig.3.(a)). Normally, the larger is, the larger the feasible region of the class is liu2016large . While, as illustrated in Fig.3.(b), if the class number increases, i.e. more new classes are injected to the same space, the constrained region of each class is going to be compact as well. ∎

4 Virtual Softmax

Naturally motivated by the property.1, a simple way to introduce a large margin between classes for achieving intra-class compactness and inter-class separability is to constrain the features with more and more additional classes. In another word, injecting additional classes is to space the original classes, resulting in a margin between these original classes. Concretely, for a given classification task (namely that the number of categories to be classified is fixed), the additionally injected classes (if inserted at the exact position) will introduce a new and more rigorous decision boundary for these original classes, compressing their intra-class distribution, and furthermore the originally overlapped decision boundaries of the adjacent categories will thus be forced to separate with each other, producing a margin between these decision boundaries and thus encouraging inter-class separability (see Fig.1 for a toy example). Therefore, injecting extra classes for feature supervision is a bold yet reasonable idea to enhance the discriminative property of features, and the objective function can be formulated as:


where is the number of extra injected classes. From Eq. 2, it can be observed that it is more possible to acquire a compact region for class quantified by the angle , since the angle is optimized above a much larger set (C+K classes)333Assume that all the K classes are injected at the exact position among original C classes..

Theoretically, the larger is, the better features are. Nevertheless, in practical cases where the available training samples are limited and the total number of classes is changeless, it is intractable to enlarge the angular margin with real and existed extra classes. Moreover, a most troublesome problem is that we can not insert the extra classes exactly between the originally adjacent categories due to the random initialization of class anchor vectors before training and the dynamic nature of parameter update during optimizing.

To address the aforementioned issues, we introduce a single and dynamic negative class into original softmax. This negative class is constructed on the basis of current training instance . Since there is no real training data belonging to this class and it is employed only as a negative category (i.e. this class is utilized only to assist the training of original C classes and have no need to be treated as a positive class), we denote it as virtual negative class and have the following formulation of our proposed Virtual Softmax:


where and is the training batch size. In the above equation, we instead require , it is a special case of Eq. 2 where . Replacing a large and fixed set of virtual classes (like classes in Eq. 2) with a single dynamic virtual class incurs nearly zero extra computational cost and memory consumption compared to original softmax.

From Eq.3, one can observe that this virtual class is tactfully inserted around the class , and particularly it is inserted at the same position with as illustrated in Fig.4.(a). It well matches our motivation that insert negative class between the originally adjacent classes and , and this negative class will never stop pushing towards until they overlap with each other due to the dynamic nature of during training procedure. Moreover, from another optimization perspective, in order to correctly classify , Virtual Softmax forces to be the largest one among inner product values. Since , the only way to make , i.e. , is to optimize , however, the original softmax is to optimize , i.e. (briefly, if have the same magnitude, the original softmax is to optimize ). Obviously, the original softmax is to make to be smaller than a certain value, while our Virtual Softmax optimizes a much more rigorous objective, i.e. zero limit , aiming to produce more compact and separable features. And based on this optimization goal, the new decision boundary of class is overlapping with the class anchor , which is more strict than softmax.

Optimization: The Virtual Softmax can be optimized with standard SGD and BP. And in backward propagation, the computation of gradients are listed as follows:


5 Discussion

Except for the above analysis that Virtual Softmax optimizes a much more rigorous objective than the original Softmax for learning discriminative features, in this section, we will give some interpretations from several other perspectives, i.e. coupling decay5.1 and feature update5.2. And we also provide the visualization of the learned features for intuitive understanding5.3.

5.1 Interpretation from Coupling Decay

In this paragraph, we give a macroscopic analysis from coupling decay which can be regarded as a regularization strategy.

Observing Eq.3, can be reformulated as: , then performing the first order Taylor Expansion for the second term, a term of shows up. Therefore, minimizing Eq.3 is to minimize to some extend, and it can be viewed as a coupling decay term, i.e. data-dependent weight decay and weight-dependent data decay. It regularizes the norm of both feature representation and the parameters in classifier layer such that improves the generalization ability of deep models by reducing over-fitting. Moreover, it is supported by some experimental results, e.g. the feature norm in Fig.2 is decreased by Virtual Softmax (e.g. from 100 to 50 in 2-D space), and the performance improvement over the original Softmax is increased when using a much wider network as in Sec.6.1 (e.g. in CIFAR100/100+, increasing the model width from t=4 to t=7, the performance improvement is rising, since increasing the dimensionality of parameters while keeping data set size constant calls for stronger regularization).

However, the reason of calling the above analysis a macroscopic one is that it coarsely throws away many other relative terms and only considers the single effect of , without taking into account the collaborative efforts of other terms. Thus there are some other phenomenons that cannot be explained well, e.g. why the inter-class angular margin is increased as shown in Fig.2

and why the confusion matrix of Virtual Softmax in Fig.

5 shows more compact intra-class distribution and separable inter-class distribution than the original Softmax. To this end, we provide another discussion below from a relatively microscopic view that how the individual data representations of Virtual Softmax and original Softmax are constrained and formed in the feature space.

5.2 Interpretation From Feature Update

Here, considering the collaborative effect of all the terms in Eq.3, we give another microscopic interpretation from the perspective of feature update, which is also a strong justification of our method. In part, it reveals the reason why the learned features by Virtual Softmax are much more compact and separable than original softmax.

Figure 4: (a) shows the virtual class by green arrow. (b) exhibits the feature layer in CNN. (c) illustrates the feature update, the blue arrow represents obtained by original softmax and the red arrow represents obtained by Virtual Softmax.

To simplify our analysis, we only consider the update in a linear feature layer, namely that the input to this linear feature layer is fixed. As illustrated in Fig.4.(b), the feature vector is computed as 444

Although the following discussion is based on this assumption, the actual uses of activation functions (like the piecewise linear function ReLU and PReLU) and the basis

do not affect the final performances., where is the input vector, is the weight parameters in this linear layer. We denote the -th element of vector as , the connected weight vector (i.e. the -th column of ) as . Thus, , after computing the partial derivative , the parameters are updated by SGD as , where is the learning rate. Since is fixed, in the next training iteration, the new feature output can be computed by the updated as:


thus from Eq. 5.2, it can be inferred that the holistic feature vector can be obtained by , where , implying that updating weight parameters with SGD can implicitly lead to the update of the output feature in the similar way. Based on this observation, putting the partial derivatives of Softmax and Virtual Softmax into Eq. 5.2 respectively, we can obtain the following corresponding updated features for Softmax (Eq. 7) and Virtual Softmax (Eq. 8) respectively:


Since the above equations are complicated enough to discuss, here for simplicity, we will give an approximate analysis. Consider a well trained feature (i.e. and ) and that the parameters in Softmax and Virtual Softmax are the same. Then, omitting the relatively smaller terms in denominators, i.e. , Eq.7, 8 can be separately approximated as:


from the above Eq.9 and Eq.10, we consider the magnitude and direction of ’vector1’, ’vector2’ and ’vector3’, and then the updated by the corresponding softmax and Virtual Softmax can be easily illustrated as in Fig.4.(c). It can be firstly observed that the norm of red arrow is smaller than that of blue arrow, showing a similar result with the analysis in Sec.5.1, i.e. Virtual Softmax will regularize the feature norm. Meanwhile, one can also observe another important phenomenon that the feature vector optimized by Virtual Softmax has a much smaller angle to the class anchor than original Softmax, well explaining the reason why the features learned by Virtual Softmax is much more compact and separable than original Softmax. In summary, by considering the collaborative effort of many terms, we could know the working principle of our Virtual Softmax better. The Virtual Softmax not only provides regularization but more importantly, intensifies the discrimination property within the learned features.

Although it is based on an approximate analysis, to some extent, it can give us an heuristic interpretation of Virtual Softmax why it is capable of encouraging the discrimination of features from a novel feature update perspective. And practically, without these assumptions the Virtual Softmax can indeed produce discriminative features, validated by the visualization in Sec.

5.3 and the experimental results in Sec. 6.

5.3 Visualization of Compactness and Separability

In order to highlight and emphasize that Virtual Softmax indeed encourages the discriminative feature learning, we provide a clear visualization of the learned features on MNIST dataset lecun1998mnist in 2-D and 3-D space respectively, as shown in Fig.2. From it, one can observe that the learned features by our Virtual Softmax are much compact and well separated, with larger inter-class angular margin and tighter intra-class distribution than softmax, and Virtual Softmax can consistently improve the performances in both 2-D (99.2% vs. 98.91%) and 3-D (99.38% vs. 99.13%) cases. Furthermore, we also visualize the leaned features in k-D space (where ) with the confusion matrix. The confusion matrix comparison between original softmax and our Virtual Softmax are shown in Fig.5. Specifically, we compute the included angle cosine (i.e. ) between any two feature vectors which are extracted from the testing splits of CIFAR10/100/100+ krizhevsky2009learning datasets. From Fig.5, it can be observed that, over the there datasets, the intra-class similarities are enhanced and the inter-class similarities are reduced when training with our Virtual Softmax. Moreover, our Virtual Softmax improves nearly , and classification accuracies over the original softmax on CIFAR10/100/100+, respectively. To summarize, all these aforementioned experiments demonstrate that our Virtual Softmax indeed serves as an efficient algorithm which has a much stronger power of enlarging inter-class angular margin and compressing intra-class distribution than softmax and thus can significantly improve the recognition performances.

5.4 Relationship to Other Methods

Layer MNIST(for fig.2) MNIST SVHN CIFAR10 CIFAR100/100+
Block1 [5x5,32

t]x2,padding 2

[3x3,32t]x4 [3x3,32t]x5 [3x3,32t]x5 [3x3,32t]x5
Pool1 Max-Pooling
Block2 [5x5,64t]x2,padding 2 [3x3,32t]x4 [3x3,64t]x4 [3x3,64t]x4 [3x3,64t]x4
Pool2 Max-Pooling
Block3 [5x5,128t]x2,padding 2 [3x3,32t]x4 [3x3,128t]x4 [3x3,128t]x4 [3x3,128t]x4
Pool3 Max-Pooling Max-Pooling Max-Pooling Max-Pooling Ave-Pooling
Fully Connected 2/3 64 64 64 512
Table 1: Model architectures for different benchmarks. [3x3,32t]x4 denotes 4 cascaded convolutional layers with filters of size 3x3. The toy models are of . Max-Pooling is with

kernel and stride of 2.

Figure 5: Cosine confusion matrixes on the test splits of CIFAR10/100/100+. On each dataset, both Softmax and Virtual Softmax use the same CNN architecture.

There are a few research works concentrating on refining the Softmax objective, e.g. L-Softmaxliu2016large , A-Softmaxliu2017sphereface and Noisy SoftmaxChen_2017_CVPR . Research works liu2016large ; liu2017sphereface require to manually select a tractable constraint, and need a careful and annealing-like training procedure which is under human control. However, our Virtual-Softmax can be easily trained end-to-end without or with little human interventions, since the margin constraint is introduced automatically by the virtual class. Moreover, our method differs from L-Softmax and A-softmax in a heuristic idea that training deep models with the additional virtual class not only the given ones, which is the first work to extend the original Softmax to employ virtual class for discriminative feature learning and may inspire other researchers. And it can also provide regularization for deep models implicitly. Method Chen_2017_CVPR aims to improve the generalization ability of DCNN by injecting noise into Softmax, which heads from a different perspective. In summary, our Virtual-Softmax comes from a clear and unusual motivation that injects a dynamic virtual class for enhancing features, which is different from the listed other methodsliu2016large ; liu2017sphereface ; Chen_2017_CVPR , even though all the methods intend to learn better features.

6 Experiments and Results

We evaluate our Virtual Softmax on classification tasks and on face verification tasks. For fair comparison, we train with the same network for both Virtual Softmax and the baseline softmax.

Small-Set Object Classification: We follow zagoruyko2016wide ; liu2016large ; Chen_2017_CVPR to devise our CNN models. Denote as the widening factor, which is used to multiply the basic number of filters in one convolutional layer, then our architecture configurations are listed in Table.1. For training, the initial learning rate is 0.1, and is divided by 10 at (20k, 27k) and (12k, 18k) in CIFAR100 and the other datasets respectively, and the corresponding total iterations are 30k and 20k.

Fine-grained Object Classification

: We fine-tune some popular models pre-trained on ImageNet

russakovsky2015imagenet , including GooglenetV1szegedy2015going and GooglenetV2szegedy2016rethinking , by replacing the last softmax layer with our Virtual Softmax. The learning rates are fixed to and for the pre-trained layers and the randomly initialized layer respectively, and stop training at 30k iteration.

Large-Set Object Classification: Here, we use the network for CIFAR100 with t=7, the learning rate starts from 0.1 and is divided by 10 at 20k, 40k, 60k iterations. The maximal iteration is 70k.

Face Verification: We employ the published Resnet model in liu2017sphereface to evaluate our Virtual Softmax. Start with a learning rate of 0.1, divide it by 10 at (30k, 50k) and stop training at 70k.

Compared Methods: We set L-Softmax(LS)liu2016large , Noisy-Softmax(NS)Chen_2017_CVPR and A-Softmax(AS)liu2017sphereface as the compared methods and re-implement them with the same experimental configurations as us.

All of our experiments are implemented by Caffe

jia2014caffe . The models are trained on one TitanX and we fill it with different batch sizes for different networks. For data preprocessing, we follow NSChen_2017_CVPR . For testing, we use original softmax to classify the testing data in classification tasks and cosine distance to evaluate the performances in face verification tasks.

6.1 Ablation Study on Network Width

Figure 6: Testing Acc(%) on CIFAR10/100/100+ datasets.
Softmax 9.84 8.2 7.76 7.15
Virtual Softmax 8.95 7.27 7.04 6.68
Improvement 0.77 0.93 0.72 0.47
Softmax 32.72 30.74 28.76 27.7
Virtual Softmax 29.84 28.4 27.81 26.02
Improvement 2.88 2.34 0.95 1.68
Softmax 30.44 28.59 27.21 25.52
Virtual Softmax 28.62 26.81 26.17 24.01
Improvement 1.82 1.78 1.04 1.51
Table 2: Recognition error rates(%) on CIFAR10/100 datasets. is the widening factor and + denotes data augmentation.

As listed in Table.1, our toy network on each dataset is with the widening factor of . We expand the toy models by setting and on CIFAR10 and CIFAR100 respectively, and the experimental results are listed in Table.2. From these results, for example on CIFAR100+, it can be observed that as the model width increasing (i.e. from to ) the recognition error rate of Virtual Softmax is diminishing, and our method achieves consistent performance gain over the original softmax when training with different networks, verifying the robustness of our method. Furthermore, when training with Virtual Softmax, the recognition error rates of wider models are consistently lower than that of thinner models across all these datasets, indicating that the Virutal Softmax does not easily suffer from over-fitting and supporting the analysis in Sec.5.1. And we plot the curves of testing Acc on CIFAR10/100/100+ as shown in Fig.6, one can observe that our Virtual Softmax can be easily optimized, with a similar convergence speed compared to the original softmax.

6.2 Evaluation on objecect datasets

MNIST lecun1998mnist , SVHN netzer2011reading , CIFAR krizhevsky2009learning are the popular used small-set object classification datasets, including different number of classes. From Table.4, it can be observed that on both MNIST and SVHN datasets the Virtual Softmax not only surpasses the original softmax using the same network (i.e. on MNIST, on SVHN) but also outperforms LSliu2016large , NSChen_2017_CVPR and ASliu2017sphereface , showing the effectiveness of our method. Moreover, we also report the experimental results on CIFAR dataset as in Table.4. Specifically, one can observe that our Virtual Softmax drastically improves nearly accuracies over the baseline softmax on CIFAR10/100/100+ respectively. Meanwhile, it outperforms all of the other methods on both CIFAR10/100 datasets, e.g. it surpasses both the RestNet-110he2016deep and Densenet-40huang2016densely on CIFAR100+ which are much deeper and more complex than our architecture, and also surpasses the listed compared methods.

Method MNIST(%) SVHN(%) Maxout goodfellow2013maxout 0.45 2.47 DSN lee2015deeply 0.39 1.92 R-CNN liang2015recurrent 0.31 1.77 WRN zagoruyko2016wide - 1.85 DisturbLabel xie2016disturblabel 0.33 2.19 Noisy Softmax Chen_2017_CVPR 0.33 - L-Softmax liu2016large 0.31 - Softmax 0.35 2.11 NS*Chen_2017_CVPR 0.32 2.04 LS*liu2016large 0.30 2.01 AS*liu2017sphereface 0.31 2.04 Virtual Softmax 0.28 1.93
Table 3: Recognition error rates on MNIST and SVHN. * denotes our reproducing.
Method CIFAR10(%) CIFAR100(%) CIFAR100+(%) GenPool lee2016generalizing 7.62 32.37 - DisturbLabel xie2016disturblabel 9.45 32.99 26.63 Noisy Softmax Chen_2017_CVPR 7.39 28.48 - L-Softmax liu2016large 7.58 29.53 - ACU jeon2017active 7.12 27.47 - ResNet-110 he2016deep - - 25.16 Densenet-40 huang2016densely 7.00 27.55 24.42 Softmax 7.15 27.7 25.52 NS*Chen_2017_CVPR 6.91 26.33 25.20 LS*liu2016large 6.77 26.18 24.32 AS*liu2017sphereface 6.83 26.09 24.11 Virtual Softmax 6.68 26.02 24.01
Table 4: Recognition error rates on CIFAR datasets. + denotes data augmentation. * denotes our reproducing.
Method CUB(%) Pose Normalization branson2014bird 75.7 Part-based RCNN zhang2014part 76.4 VGG-BGLm zhou2016fine 80.4 PG Alignment krause2015fine 82.8 Softmax (V1) 73.5 NS*Chen_2017_CVPR (V1) 74.8 LS*liu2016large (V1) 76.5 AS*liu2017sphereface (V1) 75.2 Virtual Softmax (V1) 77.1 Softmax (V2) 77.2 NS*Chen_2017_CVPR (V2) 77.9 LS*liu2016large (V2) 80.5 AS*liu2017sphereface (V2) 80.2 Virtual Softmax (V2) 81.1
Table 5: Accuracy results on CUB200. * denotes our reproducing.
Method Models LFW SLLFW DeepID2+ sun2015deeply 25 99.47 - VGG parkhi2015deep 1 97.27 88.13 Lightened CNN wu2015lightened 1 98.13 91.22 L-Softmax liu2016large 1 98.71 - Center loss wen2016discriminative 1 99.05 - Noisy Softmax Chen_2017_CVPR 1 99.18 94.50 Normface wang2017normface 1 99.19 - A-Softmax liu2017sphereface 1 99.42 - Softmax 1 99.10 94.59 NS*Chen_2017_CVPR 1 99.16 94.75 LS*liu2016large 1 99.37 95.58 AS*liu2017sphereface +Normface*wang2017normface 1 99.57 96.45 Virtual Softmax 1 99.46 95.85
Table 6: Verification results (%) on LFW/SLLFW. * denotes our reproducing.
Method Top1 Top5
Softmax 47.63 73.14
NS*Chen_2017_CVPR 47.96 73.25
LS*liu2016large 48.59 73.82
AS*liu2017sphereface 48.66 73.57
Virtual Softmax 48.84 74.06
Table 7: Acc (%) on ImageNet32

CUB200 welinder2010caltech is the popular fine-grained object classification set. The comparison results between other state-of-the-art research works and our Virtual Softmax are shown in Table.6, V1 and V2 denote the corresponding GoogleNet models. One can observe that the Virtual Softmax outperforms the baseline softmax over all the two pre-trained models, and surpasses all the compared methods LSliu2016large , NSChen_2017_CVPR and ASliu2017sphereface . Additionally, training with only the Virtual Softmax, our final result is comparable to other remarkable works which exploit many assistant attention and alignment models, showing the superiority of our method.

ImageNet32Chrabaszcz2017A : is a downsampled version of large-scale dataset ImageNet russakovsky2015imagenet , which contains exactly the same number of images as original dataset but with 32x32 size. The results are in Tab.7, one can observe that Virtual Softmax performs the best.

6.3 Evaluation on face datasets

LFW huang2007labeled is a popular face verification benchmark. SLLFW deng2017fine generalizes the original protocol in LFW to a more difficult verification task, indicating that the images are all coming from LFW but the testing pairs are more knotty to verify. For data preprocessing, we follow the method in Chen_2017_CVPR . Then augment the training data by randomly mirroring. We adopt the public available Resnet model from A-Softmaxliu2017sphereface . Then we train this Resnet model with the cleaned subset of guo2016ms . The final results are listed in Table.6. One can observe that our Virtual Softmax drastically improves the performances over the baseline softmax on both LFW and SLLFW datasets, e.g. from to and to , respectively. And it also outperforms both NSChen_2017_CVPR and LSliu2016large , showing the effectiveness of our method. Moreover, since the class number for training is very large, i.e. 67k, the optimizing procedure is difficult than that in object classification tasks, therefore, if the training phase can be eased by human guidance and control, the performance will be more better, e.g. AS*liu2017sphereface +Normface*wang2017normface achieve the best results when artificially and specifically choose the optimal margin constraints and feature scale for the current training set.

7 Conclusion

In this paper, we propose a novel but extremely simple method Virtual Softmax to enhance the discriminative property of learned features by encouraging larger angular margin between classes. It derives from a clear motivation and generalizes the optimization goal of the original softmax to a more rigorous one, i.e. zero limit . Moreover, it also has heuristic interpretations from feature update and coupling decay perspectives. Extensive experiments on both object classification and face verification tasks validate that our Virtual Softmax can significantly outperforms the original softmax and indeed serves as an efficient feature-enhancing method.

Acknowledgments: This work was partially supported by the National Natural Science Foundation of China under Grant Nos. 61573068 and 61871052, Beijing Nova Program under Grant No. Z161100004916088, and sponsored by DiDi GAIA Research Collaboration Initiative.