Feature Transformation Ensemble Model with Batch Spectral Regularization for Cross-Domain Few-Shot Classification

05/18/2020 ∙ by Bingyu Liu, et al. ∙ 0

Deep learning models often require much annotated data to obtain good performance. In real-world cases, collecting and annotating data is easy in some domains while hard in others. A practical way to tackle this problem is using label-rich datasets with large amounts of labeled data to help improve prediction performance on label-poor datasets with little annotated data. Cross-domain few-shot learning (CD-FSL) is one of such transfer learning settings. In this paper, we propose a feature transformation ensemble model with batch spectral regularization and label propagation for the CD-FSL challenge. Specifically, we proposes to construct an ensemble prediction model by performing multiple diverse feature transformations after a shared feature extraction network. On each branch prediction network of the model we use a batch spectral regularization term to suppress the singular values of the feature matrix during pre-training to improve the generalization ability of the model. The proposed model can then be fine tuned in the target domain to address few-shot classification. We also further apply label propagation and data augmentation to further mitigate the shortage of labeled data in target domains. Experiments are conducted on a number of CD-FSL benchmark tasks with four target domains and the results demonstrate the superiority of our proposed method.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Many current deep learning methods for visual recognition tasks often rely on large amounts of labeled training data to achieve high performance. Collecting and annotating such large training datasets is expensive and impractical in many cases. In order to speed up the research progress, the cross-domain few-shot learning (CD-FSL) challenge [2] has been released. It contains data from the CropDiseases, EuroSAT, ISIC2018 and ChestX datasets. The selected datasets can reflect the actual use cases for deep learning.

Meta-learning is a widely used strategy for few-shot learning. However, recent research [2] indicates that traditional “pre-training and fine-tuning” can outperform meta-learning based few-shot learning algorithms when there exists a large domain gap between source base classes and target novel classes. Nevertheless, the capacity of fine-tuning can still be limited when facing the large domain gap. To tackle this problem, in the paper, we propose a batch spectral regularization (BSR) mechanism to suppress all the singular values of the feature matrix in pre-training so that the pre-trained model can avoid overfitting to the source domain and generalize well to the target domain. Moreover, we propose a feature transformation ensemble model that builds multiple predictors in projected diverse feature spaces to facilitate cross-domain adaptation and increase prediction robustness. To mitigate the shortage of labeled data in the target domain, we exploit the unlabeled query set in fine-tuning through entropy minimization. We also apply a label propagation (LP) step to refine the original classification results, and exploit data augmentation techniques to augment both the few-shot and test instances from different angles to improve prediction performance. Experiments are conducted on the CD-FSL benchmark tasks and the results demonstrate the superiority of our proposed model over the strong fine-tuning baseline.

2 Approach

Figure 1: An overview of the proposed approach.

In the cross-domain few-shot learning setting, we have a source domain and a target domain . We use all the labeled data in the source domain for pre-training. In the target domain, a numbe of -way -shot classification tasks are sampled, each with a support set composed of labeled examples from novel classes. The labeled support set can be used to fine-tune the pre-trained model, while a query set from the same classes is used to evaluate the model performance. Inspired by the ensemble networks for zero-shot learning [4], we build an ensemble prediction model for cross-domain few-shot learning. The overview of our proposed ensemble model is depicted in Fig. 1. Below we introduce the components involved in the model.

2.1 Feature Transformation Ensemble Model

We build the ensemble model by increasing the diversity of the feature representation space while maintaining the usage of the entire training data for each prediction branch network. As shown in Fig. 1

, we use a Convolutional Neural Network (CNN)

to extract advanced visual features from the input data, and then transform the features into multiple diverse feature representation spaces using different random orthogonal projection matrices on different branches. Each projection matrix is generated in the follow way. We randomly generate a symmetric matrix

, and then form the orthogonal matrix

using the eigenvectors of

such that , where represents the eigenvector corresponding to the top

-th eigenvalue of

. With each projection matrix , we can transform the extracted features into a new feature representation space such that , and build a soft-max predictor in this feature space. By using randomly generated orthogonal projection matrices, we can then build classifiers. In the pre-training stage in the source domain, all the labeled source data is used to train each branch network, which includes the composite feature extractor , and the classifier , by minimizing the cross-entropy loss. In a training batch

, the loss function can be written as


where denotes the cross-entropy loss function. After pre-training in the source domain, the fine-tuning on the labeled support set of the target domain can be conducted in the similar way, while the testing on the query instances can be naturally produced in an ensemble manner by taking the average of the classifiers’ prediction results.

2.2 Batch Spectral Regularization

Previous work [1] shows that penalizing smaller singular values of a feature matrix can help mitigate negative transfer in fine-tuning. We extend this penalizer into the full spectrum and propose a batch spectral regularization (BSR) mechanism to suppress all the singular values of the batch feature matrices in pre-training, aiming to avoid overfitting to the source domain and increase generalization ability to the target domain. This regularization is applied for each branch network of the ensemble model separately in the same way. For simplicity, we omitted the branch network index in the following presentation.

Specifically, for a stochastic gradient descent based training algorithm, we work with training batches. Given a batch of training instances

, its feature matrix can be obtained as , where is the batch size and

is the feature vector for the

-th instance in the batch. The BSR term can then be written as


where are singular values of the batch feature matrix . The spectral regularized training loss for each batch will be:


2.3 Label Propagation

Due to the lack of labeled data in the target domain, the model fine-tuned with the support set can can easily make wrong predictions on the query instances. Following the effective label refinement procedure in [3], we propose to apply a label propagation (LP) step to exploit the semantic information of unlabeled test data in the extracted feature space and refine the original classification results.

Given the prediction score matrix on the query instances with the fine-tuned classifier , we keep the top- fraction of scores in each class (the columns of ) and set other values to zeros in order to propagate only the most confident predictions. We then build a k-NN graph over the query instances based on the extracted features . We use the squared Euclidean distance between each pair of images such as

to determine the k-NN graph. The RBF kernel based affinity matrix

can be computed as follows:


where is the radius of the RBF kernel and denotes the k-nearest neighbors of the -th image. The normalized Laplacian matrix can then be calculated as , where is a diagonal matrix with . The label propagation is then performed to provide the following refined prediction score matrix:



is an identity matrix and

is a trade-off parameter. After LP, is used as the predicted class for the -th image.

2.4 Entropy Minimization

We extend the semi-supervised learning mechanism into the fine-tuning phase in the target domain by minimizing the prediction entropy on the unlabeled query set:


where denotes a query batch. We can add this term to the original cross-entropy loss on the support set batch and form a transductive fine-tuning loss function:


where is a trade-off parameter.

2.5 Data Augmentation

We also exploit data augmentation (DA) strategy with several random operations to supplement the support set and make the models learn with more variations. In particular, we use combinations of some operations such as image scaling, random crop, random flip, random rotation and color jitter to generate a few variants for each image. The fine-tuning can be conducted on the augmented support set. The same augmentation can be conducted for the query set as well, where several variants of each image can be generated to share the same label. Thus the prediction result on each image can be determined by averaging the prediction results on all the augmented variants of the same image.

3 Experiments

Augmentation Hyper-parameters
Scale (S)
RandomResizedCrop (C)
ImageJitter (J)
Brightness: 0.4
Color: 0.4
RandomHorizontalFlip (H)

Flip probability: 50%

RandomRotation (R) degrees
Table 1: Hyper-parameters of augmentation operations.
Dataset Augmentation
S + SJHR + SR + SJ + SH
& CropDiseases
ChestX S + SJH + C + CJ + CH
Table 2: Compound modes of augmentation operations.
5-way 5-shot 5-way 20-shot 5-way 50-shot 5-way 5-shot 5-way 20-shot 5-way 50-shot
Fine-tuning [2] 25.97%0.41% 31.32%0.45% 35.49%0.45% 48.11%0.64% 59.31%0.48% 66.48%0.56%
BSR 26.84%0.44% 35.63%0.54% 40.18%0.56% 54.42%0.66% 66.61%0.61% 71.10%0.60%
BSR+LP 27.10%0.45% 35.92%0.55% 40.56%0.56% 55.86%0.66% 67.48%0.60% 72.17%0.58%
BSR+DA 28.20%0.46% 36.72%0.51% 42.08%0.53% 54.97%0.68% 66.43%0.57% 71.62%0.60%
BSR+LP+ENT 26.86%0.45% 35.60%0.51% 42.26%0.53% 56.82%0.68% 68.97%0.56% 74.13%0.56%
BSR+LP+DA 28.50%0.48% 36.95%0.52% 42.32%0.53% 56.25%0.69% 67.31%0.57% 72.33%0.58%
BSR (Ensemble) 28.44%0.45% 37.05%0.50% 43.22%0.54% 55.47%0.68% 68.00%0.59% 73.36%0.54%
BSR+LP (Ensemble) 28.66%0.44% 37.44%0.51% 43.72%0.54% 57.14%0.67% 68.99%0.58% 74.62%0.54%
BSR+DA (Ensemble) 29.09%0.45% 37.89%0.53% 43.98%0.56% 56.13%0.66% 67.10%0.61% 73.16%0.54%
BSR+LP+ENT (Ensemble) 28.00%0.46% 36.87%0.52% 43.79%0.53% 58.02%0.68% 70.22%0.59% 74.94%0.55%
BSR+LP+DA (Ensemble) 29.72%0.45% 38.34%0.53% 44.43%0.56% 57.40%0.67% 68.09%0.60% 74.08%0.55%
EuroSAT CropDiseases
5-way 5-shot 5-way 20-shot 5-way 50-shot 5-way 5-shot 5-way 20-shot 5-way 50-shot
Fine-tuning [2] 79.08%0.61% 87.64%0.47% 90.89%0.36% 89.25%0.51% 95.51%0.31% 97.68%0.21%
BSR 80.89%0.61% 90.44%0.40% 93.88%0.31% 92.17%0.45% 97.90%0.22% 99.05%0.14%
BSR+LP 84.35%0.59% 91.99%0.37% 95.02%0.27% 94.45%0.40% 98.65%0.19% 99.38%0.11%
BSR+DA 82.75%0.55% 92.61%0.31% 95.26%0.39% 93.99%0.39% 98.62%0.15% 99.42%0.08%
BSR+LP+ENT 85.70%0.53% 92.90%0.33% 95.40%0.29% 95.69%0.35% 98.60%0.18% 99.27%0.12%
BSR+LP+DA 85.97%0.52% 93.73%0.29% 96.07%0.30% 95.97%0.33% 99.10%0.12% 99.66%0.07%
BSR (Ensemble) 83.93%0.53% 92.55%0.33% 95.11%0.24% 93.54%0.41% 98.34%0.20% 99.22%0.12%
BSR+LP (Ensemble) 86.08%0.55% 93.81%0.30% 95.97%0.23% 95.48%0.38% 98.94%0.16% 99.49%0.11%
BSR+DA (Ensemble) 85.19%0.51% 93.68%0.28% 96.14%0.26% 94.80%0.36% 98.69%0.16% 99.51%0.09%
BSR+LP+ENT (Ensemble) 87.17%0.52% 93.96%0.29% 96.09%0.22% 96.04%0.36% 98.94%0.16% 99.45%0.10%
BSR+LP+DA (Ensemble) 88.13%0.49% 94.72%0.28% 96.89%0.19% 96.59%0.31% 99.16%0.14% 99.73%0.06%
Table 3: Results on the CD-FSL benchmark.
Methods Average
Fine-tuning [2] 67.23%0.46%
BSR 70.76%0.46%
BSR+LP 71.91%0.44%
BSR+DA 71.89%0.44%
BSR+LP+ENT 72.68%0.42%
BSR+LP+DA 72.85%0.42%
BSR (Ensemble) 72.35%0.43%
BSR+LP (Ensemble) 73.36%0.42%
BSR+DA (Ensemble) 72.95%0.42%
BSR+LP+ENT (Ensemble) 73.62%0.42%
BSR+LP+DA (Ensemble) 73.94%0.40%
Table 4: Averages across all datasets and shot levels.

3.1 Setup

In the experiments, we use the evaluation protocol in the CD-FSL challenge [2]

, which takes 15 images from each class as the query set and uses 600 randomly sampled few-shot episodes in each target domain, The average accuracy and 95% confidence interval are reported.

As for the model architecture, we use ResNet-10 as the CNN feature extractor and a fully-connected layer with soft-max activation as the classifier . We set the trade-off parameters , and the number of branches . For the label propagation step, we use for the k-NN graph construction, and set as the average of the squared distances of the edges in the k-NN graph. The parameters and

are set to 0.2 and 0.5 respectively. We adopt mini-batch SGD with momentum of 0.9 for both pre-training and fine-tuning. During the pre-training stage, models are trained for 400 epochs. The learning rate and the weight decay are set to 0.001 and 0.0005 respectively. During fine-tuning, we set the learning rate to 0.01 and fine-tune for 100 epochs.

For data augmentation (DA), we choose 5 types of augmentation operations as shown in Table 1. We use 5 compound modes of these operations to generate data in different target domains. The specific operations used in each target domain are shown in Table 2.

3.2 Results

We investigate a number of variants of the proposed model by comparing with the strong fine-tuning baseline result reported in [2]. We first investigate a single prediction network with batch spectral regularization (BSR) without ensemble and its other variants that further incorporate label propagation (LP) or/and data augmentation (DA). Then we extend these variants into the ensemble model framework with . The results are reported in Table 3, and the average accuracies (and 95% confidence internals) across all datasets and shot levels are shown in Table 4.

We can see that even with only BSR, the proposed method can already significantly outperform the fine-tuning baseline (average 70.76% vs 67.23%). The ensemble BSR further improves the results (72.35%). The LP and DA components can also help improve the CD-FSL performance. We observe that on target domains more similar to the source domain, LP performs better than DA and vice versa. This shows that LP and DA focus on different aspects of the data. As a result combining LP and DA can further improve the performances. Moreover, DA is not very effective for the ISIC domain, where it even degrades the performance in some cases. Also the experiment’s running time is typically longer with DA. By replacing DA with ENT, we can obtain similar overall performance. With a single model, the best average result achieved by BSR+LP+DA is 72.85%, while BSR+LP+ENT achieves 72.68%. With the ensemble model, BSR+LP+DA (Ensemble) produces the best average result 73.94%, while BSR+LP+ENT (Ensemble) yields 73.62%.

4 Conclusion

In this paper, we proposed a feature transformation based ensemble model for CD-FSL. The model also incorporates batch spectral regularization in pre-training, and exploits data augmentation and label propagation during fine-tuning and testing in the target domain. The combinational models produced superior CD-FSL results comparing to the strong fine-tuning baseline.


  • [1] X. Chen, S. Wang, B. Fu, M. Long, and J. Wang (2019) Catastrophic forgetting meets negative transfer: batch spectral shrinkage for safe transfer learning. In NeurIPS, Cited by: §2.2.
  • [2] Y. Guo, N. C. Codella, L. Karlinsky, J. R. Smith, T. Rosing, and R. Feris (2019) A new benchmark for evaluation of cross-domain few-shot learning. arXiv preprint arXiv:1912.07200. Cited by: §1, §1, §3.1, §3.2, Table 3, Table 4.
  • [3] M. Ye and Y. Guo (2017)

    Labelless scene classification with semantic matching.

    In BMVC, Cited by: §2.3.
  • [4] M. Ye and Y. Guo (2019) Progressive ensemble networks for zero-shot recognition. In CVPR, Cited by: §2.