Transductive Propagation Network for Few-shot Learning

05/25/2018 ∙ by Yanbin Liu, et al. ∙ AItrics University of Oxford University of Technology Sydney 0

Few-shot learning aims to build a learner that quickly generalizes to novel classes even when a limited number of labeled examples (so-called low-data problem) are available. Meta-learning is commonly deployed to mimic the test environment in a training phase for good generalization, where episodes (i.e., learning problems) are manually constructed from the training set. This framework gains a lot of attention to few-shot learning with impressive performance, though the low-data problem is not fully addressed. In this paper, we propose Transductive Propagation Network (TPN), a transductive method that classifies the entire test set at once to alleviate the low-data problem. Specifically, our proposed network explicitly learns an underlying manifold space that is appropriate to propagate labels from few-shot examples, where all parameters of feature embedding, manifold structure, and label propagation are estimated in an end-to-end way on episodes. We evaluate the proposed method on the commonly used miniImageNet and tieredImageNet benchmarks and achieve the state-of-the-art or promising results on these datasets.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Recent breakthroughs in deep learning 

(Krizhevsky et al., 2012; Simonyan and Zisserman, 2015; He et al., 2016) highly rely on the availability of large amounts of labeled data. However, this reliance on large data increases the burden of data collection, which hinders its potential applications to the low-data regime where the labeled data is rare and difficult to gather. On the contrary, humans have the ability to recognize new objects after observing only one or few instances (Lake et al., 2011). For example, children can generalize the concept of “apple” after given a single instance of it. This significant gap between human and deep learning has reawakened the research interest on few-shot learning (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Ravi and Larochelle, 2017; Lee and Choi, 2018; Xu et al., 2017; Wang et al., 2018).

Few-shot learning aims to learn a classifier that generalizes well with a few examples of each of these classes. Traditional techniques such as fine-tuning (Jia et al., 2014) that work well with deep learning models would severely overfit on this task (Vinyals et al., 2016; Finn et al., 2017)

, since a single or only a few labeled instances would not accurately represent the true data distribution and will result in learning classifiers with high variance, which will not generalize well to new data.

In order to solve this overfitting problem,  Vinyals et al. (2016) proposed a meta-learning strategy which learns over diverse classification tasks over large number of episodes rather than only on the target classification task. In each episode, the algorithm learns the embedding of the few labeled examples (the support set), which can be used to predict classes for the unlabeled points (the query set) by distance in the embedding space. The purpose of episodic training is to mimic the real test environment containing few-shot support set and unlabeled query set. The consistency between training and test environment alleviates the distribution gap and improves generalization. This episodic meta-learning strategy, due to its generalization performance, has been adapted by many follow-up work on few-shot learning. Finn et al. (2017) learned a good initialization that can adapt quickly to the target tasks. Snell et al. (2017) used episodes to train a good representation and predict classes by computing Euclidean distance with respect to class prototypes.

Although episodic strategy is an effective approach for few-shot learning as it aims at generalizing to unseen classification tasks, the fundamental difficulty with learning with scarce data remains for a novel classification task. One way to achieve larger improvements with limited amount of training data is to consider relationships between instances in the test set and thus predicting them as a whole, which is referred to as transduction, or transductive inference. In previous work (Joachims, 1999; Zhou et al., 2004; Vapnik, 1999), transductive inference has shown to outperform inductive methods which predict test examples one by one, especially in small training sets. One popular approach for transduction is to construct a network on both the labeled and unlabeled data, and propagate labels between them for joint prediction. However, the main challenge with such label propagation (and transduction) is that the label propagation network is often obtained without consideration of the main task, since it is not possible to learn them at the test time.

Yet, with the meta-learning by episodic training, we can learn the label propagation network as the query examples sampled from the training set can be used to simulate the real test set for transductive inference. Motivated by this finding, we propose Transductive Propagation Network (TPN) to deal with the low-data problem. Instead of applying the inductive inference, we utilize the entire query set for transductive inference (see Figure 1

). Specifically, we first map the input to an embedding space using a deep neural network. Then a graph construction module is proposed to exploit the manifold structure of the novel class space using the union of support set and query set. According to the graph structure, iterative label propagation is applied to propagate labels from the support set to the query set and finally leads to a closed-form solution. With the propagated scores and ground truth labels of the query set, we compute the cross-entropy loss with respect to the feature embedding and graph construction parameters. Finally, all parameters can be updated end-to-end using backpropagation.

Figure 1: A conceptual illustration of our transductive meta-learning framework, where lines between nodes represent graph connections and their colors represent the potential direction of label propagation. The neighborhood graph is episodic-wisely trained for transductive inference.

The main contribution of this work is threefold.

  • To the best of our knowledge, we are the first to model transductive inference explicitly in few-shot learning. Although Nichol et al. (2018)

    experimented with a transductive setting, they only share information between test examples by batch normalization rather than directly proposing a transductive model.

  • In transductive inference, we propose to learn to propagate labels

    between data instances for unseen classes via episodic meta-learning. This learned label propagation graph is shown to significantly outperform naive heuristic-based label propagation methods 

    (Zhou et al., 2004).

  • We evaluate our approach on two benchmark datasets for few-shot learning, namely miniImageNet  and tieredImageNet. The experimental results show that our Transductive Propagation Network

    outperforms the state-of-the-art methods on both datasets. Also, with semi-supervised learning, our algorithm achieves even higher performance, outperforming all semi-supervised few-shot learning baselines.

2 Related work

Meta-learning

In recent works, few-shot learning often follows the idea of meta-learning (Schmidhuber, 1987; Thrun and Pratt, 2012). Meta-learning tries to optimize over batches of tasks rather than batches of data points. Each task corresponds to a learning problem, obtaining good performance on these tasks helps to learn quickly and generalize well to the target few-shot problem without suffering from overfitting. The well-known MAML approach (Finn et al., 2017) aims to find more transferable representations with sensitive parameters. A first-order meta-learning approach named Reptile is proposed by Nichol et al. (2018). It is closely related to first-order MAML but does not need a training-test split for each task. Compared with the above methods, our algorithm has a closed-form solution for label propagation on the query points, thus avoiding gradient computation in the inner updateand usually performs more efficiently.

Embedding and metric learning approaches

Another category of few-shot learning approach aims to optimize the transferable embedding using metric learning approaches. Matching networks (Vinyals et al., 2016) produce a weighted nearest neighbor classifier given the support set and adjust feature embedding according to the performance on the query set. Prototypical networks (Snell et al., 2017) first compute a class’s prototype to be the mean of its support set in the embedding space. Then the transferability of feature embedding is evaluated by finding the nearest class prototype for embedded query points. An extension of prototypical networks is proposed in Ren et al. (2018) to deal with semi-supervised few-shot learning. Relation Network (Sung et al., 2018) learns to learn a deep distance metric to compare a small number of images within episodes. Our proposed method is similar to these approaches in the sense that we all focus on learning deep embeddings with good generalization ability. However, our algorithm assumes a transductive setting, in which we utilize the union of support set and query set to exploit the manifold structure of novel class space by using episodic-wise parameters.

Transduction

The setting of transductive inference was first introduced by Vapnik (Vapnik, 1999)

. Transductive Support Vector Machines (TSVMs) 

(Joachims, 1999) is a margin-based classification method that minimizes errors of a particular test set. It shows substantial improvements over inductive methods, especially for small training sets. Another category of transduction methods involves graph-based methods (Zhou et al., 2004; Wang and Zhang, 2006; Rohrbach et al., 2013; Fu et al., 2015). Label propagation is used in Zhou et al. (2004) to transfer labels from labeled to unlabeled data instances guided by the weighted graph. Label propagation is sensitive to variance parameter , so Linear Neighborhood Propagation (LNP) (Wang and Zhang, 2006) constructs approximated Laplacian matrix to avoid this issue. In Zhu and Ghahramani (2002), minimum spanning tree heuristic and entropy minimization are used to learn the parameter . In all these prior work, the graph construction is done on a pre-defined feature space using manually selected hyperparamters since it is not possible to learn them at test time. Our approach, on the other hand, is able to learn the graph construction network since it is a meta-learning framework with episodic training, where at each episode we simulate the test set with a subset of the training set.

In few-shot learning, Nichol et al. (2018) experiments with a transductive setting and shows improvements. However, they only share information between test examples via batch normalization (Ioffe and Szegedy, 2015) rather than explicitly model the transductive setting as in our algorithm.

3 Main approach

In this section, we introduce the proposed algorithm that utilizes the manifold structure of the given few-shot classification task to improve the performance.

3.1 Problem definition

We follow the episodic paradigm (Vinyals et al., 2016) that effectively trains a meta-learner for few-shot classification tasks, which is commonly employed in various literature (Snell et al., 2017; Finn et al., 2017; Nichol et al., 2018; Sung et al., 2018; Mishra et al., 2018). Given a relatively large labeled dataset with a set of classes , the objective of this setting is to train classifiers for an unseen set of novel classes , for which only a few labeled examples are available.

Specifically, in each episode, a small subset of classes are sampled from to construct a support set and a query set. The support set contains examples from each of the classes (i.e., -way -shot setting) denoted as , while the query set includes different examples from the same classes. Here, the support set in each episode serves as the labeled training set on which the model is trained to minimize the loss of its predictions for the query set . This procedure mimics training classifiers for and goes episode by episode until convergence.

Meta-learning implemented by the episodic training reasonably performs well to few-shot classification tasks. Yet, due to the lack of labeled instances ( is usually very small) in the support set, we observe that a reliable classifier is still difficult to be obtained. This motivates us to consider a transductive setting that utilizes the whole query set for the prediction rather than predicting each example independently. Taking the entire query set into account, we can alleviate the low-data problem and provide more reliable generalization property.

3.2 Transductive Propagation Network (TPN)

Figure 2: The overall framework of our algorithm in which the manifold structure of the entire query set helps to learn better decision boundary. The proposed algorithm is composed of four components: feature embedding, graph construction, label propagation, and loss generation.

We introduce Transductive Propagation Network (TPN) illustrated in Figure 2

, which consists of four components: feature embedding with a convolutional neural network; graph construction that produces example-wise parameters to exploit the manifold structure; label propagation that spreads labels from the support set

to the query set ; a loss generation step that computes a cross-entropy loss between propagated labels and the ground-truths on to jointly train all parameters in the framework.

3.2.1 Feature embedding

We employ a convolutional neural network to extract features of an input , where refers to the feature map and indicates a parameter of the network. Despite the generality, we adopt the same architecture used in several recent works (Snell et al., 2017; Sung et al., 2018; Vinyals et al., 2016). By doing so, we can provide more fair comparisons in the experiments, highlighting the effects of transductive approach. The network is made up of four convolutional blocks where each block begins with a 2D convolutional layer with a kernel and filter size of . Each convolutional layer is followed by a batch-normalization layer (Ioffe and Szegedy, 2015)

, a ReLU nonlinearity and a

max-pooling layer. We use the same embedding function for both the support set and the query set .

3.2.2 Graph construction

Manifold learning (Chung and Graham, 1997; Zhou et al., 2004; Yang et al., 2016) discovers the embedded low-dimensional subspace in the data, where it is critical to choose an appropriate neighborhood graph. A common choice is Gaussian similarity function:

(1)

where is a distance measure (e.g., Euclidean distance) and is the length scale parameter. The neighborhood structure behaves differently with respect to various , which means that it needs to carefully select the optimal for the best performance of label propagation (Wang and Zhang, 2006; Zhu and Ghahramani, 2002). In addition, we observe that there is no principled way to tune the scale parameter in meta-learning framework, though there exist some heuristics for dimensionalty reduction methods (Zelnik-Manor and Perona, 2004; Sugiyama, 2007).

Example-wise length-scale parameter

To obtain a proper neighborhood graph in meta-learning, we propose a graph construction module built on the union set of support set and query set: . This module is composed of a convolutional neural network which takes the feature map for to produce an example-wise length-scale parameter . Note that the scale parameter is determined example-wisely and learned in an episodic training procedure, which adapts well to different tasks and makes it suitable for few-shot learning. With the example-wise , our similarity function is then defined as follows:

(2)

where for all instances in . We only keep the -max values in each row of to construct a -nearest neighbour graph. Then we apply the normalized graph Laplacians (Chung and Graham, 1997) on , that is, , where is a diagonal matrix with its -value to be the sum of the -th row of .

Figure 3: Detailed architecture of the graph construction module, in which the length-scale parameter is example-wisely determined.
Graph construction structure

The structure of the proposed graph construction module is shown in Figure 3

. It is composed of two convolutional blocks and two fully-connected layers, where each block contains a 3-by-3 convolution, batch normalization, ReLU activation, followed by 2-by-2 max pooling. The number of filters in each convolutional block is 64 and 1, respectively. To provide an example-wise scaling parameter, the activation map from the second convolutional block is transformed into a scalar by two fully-connected layers in which the number of neurons is 8 and 1, respectively.

Graph construction in each episode

We follow the episodic paradigm for few-shot meta-learner training. This means that the graph is individually constructed for each task in each episode, as shown in Figure 1. Typically, in 5-way 5-shot training, , the dimension of is only , which is quite efficient.

3.2.3 Label propagation

We now describe how to get predictions for the query set using label propagation, before the last cross-entropy loss step. Let denote the set of matrix with nonnegative entries. We define a label matrix with if is from the support set and labeled as , otherwise . Starting from , label propagation iteratively determines the unknown labels of instances in the union set according to the graph structure using the following formulation:

(3)

where denotes the predicted labels at the timestamp , denotes the normalized weight, and controls the amount of propagated information. It is well known that the sequence has a closed-form solution as follows:

(4)

where

is the identity matrix

(Zhou et al., 2004). We directly utilize this result for the label propagation, making a whole episodic meta-learning procedure more efficient in practice.

Time complexity

Matrix inversion originally takes time complexity, which is inefficient for large . However, in our setting, (80 for 1-shot and 100 for 5-shot) is very small. Moreover, there is plenty of prior work on the scalability and efficiency of label propagation, such as Liang and Li (2018); Fujiwara and Irie (2014), which can extend our work to large-scale data. More discussions are presented in A.4

3.2.4 Classification loss generation

The objective of this step is to compute the classification loss between the predictions of the union of support and query set via label propagation and the ground-truths. We compute the cross-entropy loss between predicted scores and ground-truth labels from to learn all parameters in an end-to-end fashion, where is converted to probabilistic score using softmax:

(5)

Here, denotes the final predicted label for th instance in the union of support and query set and denotes the

th component of predicted label from label propagation. Then the loss function is computed as:

(6)

where means the ground-truth label of and is an indicator function, if is true and otherwise.

Note that in Equation (6), the loss is dependent on two set of parameters , (even though the dependency is implicit through ). All these parameters are jointly updated by the episodic training in an end-to-end manner.

4 Experiments

We evaluate and compare our TPN with state-of-the-art approaches on two datasets, i.e., miniImageNet (Ravi and Larochelle, 2017) and tieredImageNet (Ren et al., 2018). The former is the most popular few-shot learning benchmark and the latter is a much larger dataset released recently for few-shot learning.

4.1 Datasets

miniImageNet. The miniImageNet dataset is a collection of Imagenet (Krizhevsky et al., 2012) for few-shot image recognition. It is composed of 100 classes randomly selected from Imagenet with each class containing 600 examples. In order to directly compare with state-of-the-art algorithms for few-shot learning, we rely on the class splits used by  Ravi and Larochelle (2017), which includes 64 classes for training, 16 for validation, and 20 for test. All images are resized to pixels.

tieredImageNet. Similar to miniImageNet , tieredImageNet (Ren et al., 2018) is also a subset of Imagenet (Krizhevsky et al., 2012), but it has a larger number of classes from ILSVRC-12 (608 classes rather than 100 for miniImageNet). Different from miniImageNet, it has a hierarchical structure of broader categories corresponding to high-level nodes in Imagenet. The top hierarchy has 34 categories, which are divided into 20 training (351 classes), 6 validation (97 classes) and 8 test (160 classes) categories. The average number of examples in each class is 1281. This high-level split strategy ensures that the training classes are distinct from the test classes semantically. This is a more challenging and realistic few-shot setting since there is no assumption that training classes should be similar to test classes. Similarly, all images are resized to pixels.

4.2 Experimental Setup

For fair comparison with other methods, we adopt a widely-used CNN (Finn et al., 2017; Snell et al., 2017) as the feature embedding function (Section 3.2.1). The hyper-parameter of -nearest neighbour graph (Section 3.2.2) is set to and of label propagation is set to , as suggested in Zhou et al. (2004).

Following Snell et al. (2017), we adopt the episodic training procedure, i.e, we sample a set of -way -shot training tasks to mimic the -way -shot test problems. Moreover, Snell et al. (2017) proposed a “Higher Way ” training strategy which used more training classes in each episode than test case. However, we find that it is beneficial to train with more examples than test phase (Appendix A.1). This is denoted as “Higher Shot” in our experiments. For 1-shot and 5-shot test problem, we adopt 5-shot and 10-shot training respectively. In all settings, the query number is set to 15 and the performance are averaged over 600 randomly generated episodes from the test set.

All our models were trained with Adam (Kingma and Ba, 2015) and an initial learning rate of . For miniImageNet, we cut the learning rate in half every episodes and for tieredImageNet, we cut the learning rate every episodes. The reason for larger decay step is that tieredImageNet has more classes and more examples in each class which needs larger training iterations. We ran the training process until the validation loss reached a plateau.

4.3 Few-shot Learning Results

We compare our method with several state-of-the-art approaches in various settings. Even though the transductive method has never been used explicitly, batch normalization layer was used transductively to share information between test examples. For example, in Finn et al. (2017); Nichol et al. (2018), they use the query batch statistics rather than global BN parameters for the prediction, which leads to performance gain in the query set. Besides, we propose two simple transductive methods as baselines that explicitly utilize the query set. First, we propose the MAML+Transduction with slight modification of loss function to: for transductive inference. The additional term serves as transductive regularization. Second, the naive heuristic-based label propagation methods (Zhou et al., 2004) is proposed to explicitly model the transductive inference.

5-way Acc 10-way Acc
Model Transduction 1-shot 5-shot 1-shot 5-shot
MAML (Finn et al., 2017) BN 48.70 63.11 31.27 46.92
MAML+Transduction Yes 50.83 66.19 31.83 48.23
Reptile (Nichol et al., 2018) No 47.07 62.74 31.10 44.66
Reptile + BN (Nichol et al., 2018) BN 49.97 65.99 32.00 47.60
PROTO NET (Snell et al., 2017) No 46.14 65.77 32.88 49.29
PROTO NET (Higher Way) (Snell et al., 2017) No 49.42 68.20 34.61 50.09
RELATION NET (Sung et al., 2018) BN 51.38 67.07 34.86 47.94
Label Propagation Yes 52.31 68.18 35.23 51.24
TPN Yes 53.75 69.43 36.62 52.32
TPN (Higher Shot) Yes 55.51 69.86 38.44 52.77
  • * “Higher Way” means using more classes in training episodes. “Higher Shot” means using more shots in training episodes. “BN” means information is shared among test examples using batch normalization.

  • Due to space limitation, we report the accuracy with 95% confidence intervals in Appendix.

Table 1: Few-shot classification accuracies on miniImageNet. All results are averaged over 600 test episodes. Top results are highlighted.
5-way Acc 10-way Acc
Model Transduction 1-shot 5-shot 1-shot 5-shot
MAML (Finn et al., 2017) BN 51.67 70.30 34.44 53.32
MAML + Transduction Yes 53.23 70.83 34.78 54.67
Reptile (Nichol et al., 2018) No 48.97 66.47 33.67 48.04
Reptile + BN (Nichol et al., 2018) BN 52.36 71.03 35.32 51.98
PROTO NET (Snell et al., 2017) No 48.58 69.57 37.35 57.83
PROTO NET (Higher Way) (Snell et al., 2017) No 53.31 72.69 38.62 58.32
RELATION NET (Sung et al., 2018) BN 54.48 71.31 36.32 58.05
Label Propagation Yes 55.23 70.43 39.39 57.89
TPN Yes 57.53 72.85 40.93 59.17
TPN (Higher Shot) Yes 59.91 73.30 44.80 59.44
  • * “Higher Way” means using more classes in training episodes. “Higher Shot” means using more shots in training episodes. “BN” means information is shared among test examples using batch normalization.

  • Due to space limitation, we report the accuracy with 95% confidence intervals in Appendix.

Table 2: Few-shot classification accuracies on tieredImageNet. All results are averaged over 600 test episodes. Top results are highlighted.

Experimental results are shown in Table 1 and Table2. Transductive batch normalization methods tend to perform better than pure inductive methods except for the “Higher Way” PROTO NET. Label propagation without learning to propagate outperforms other baseline methods in most cases, which verifies the necessity of transduction. The proposed TPN achieves the state-of-the-art results and surpasses all the others with a large margin even when the model is trained with regular shots. When “Higher Shot” is applied, the performance of TPN continues to improve especially for 1-shot case. This confirms that our model effectively finds the episodic-wise manifold structure of test examples through learning to construct the graph for label propagation.

Another observation is that the advantages of 5-shot classification is less significant than that of 1-shot case. For example, in 5-way miniImageNet , the absolute improvement of TPN over published state-of-the-art is 4.13% for 1-shot and 1.66% for 5-shot. To further investigate this, we experimented 5-way -shot () experiments. The results are shown in Figure 4. Our TPN performs consistently better than other methods with varying shots. Moreover, it can be seen that TPN outperforms other methods with a large margin in lower shots. With the shot increase, the advantage of transduction narrows since more labelled data are used. This finding agrees with the results in TSVM (Joachims, 1999): when more training data are available, the bonus of transductive inference will be decreased.

Figure 4: 5-way performance with various training/test shots.

4.4 Comparison with semi-supervised few-shot learning

Model 1-shot 5-shot 1-shot w/D 5-shot w/D
Soft -Means (Ren et al., 2018) 50.09 64.59 48.70 63.55
Soft -Means+Cluster (Ren et al., 2018) 49.03 63.08 48.86 61.27
Masked Soft -Means (Ren et al., 2018) 50.41 64.39 49.04 62.96
TPN-semi 52.78 66.42 50.43 64.95
  • * “w/D” means with distraction. In this setting, many of the unlabelled data are from the so-called distraction classes , which is different from the classes of labelled data.

  • Due to space limitation, we report the accuracy with 95% confidence intervals in Appendix.

Table 3: Semi-supervised comparison on miniImageNet.
Model 1-shot 5-shot 1-shot w/D 5-shot w/D
Soft -Means (Ren et al., 2018) 51.52 70.25 49.88 68.32
Soft -Means+Cluster (Ren et al., 2018) 51.85 69.42 51.36 67.56
Masked Soft -Means (Ren et al., 2018) 52.39 69.88 51.38 69.08
TPN-semi 55.74 71.01 53.45 69.93
  • * “w/D” means with distraction. In this setting, many of the unlabelled data are from the so-called distraction classes , which is different from the classes of labelled data.

  • Due to space limitation, we report the accuracy with 95% confidence intervals in Appendix.

Table 4: Semi-supervised comparison on tieredImageNet.

The main difference of traditional semi-supervised learning and transduction is the source of unlabeled data. Transductive methods directly use test set as unlabeled data while semi-supervised learning usually has an extra unlabeled set. In order to compare with semi-supervised methods, we propose a semi-supervised version of TPN, named TPN-semi, which classifies one test example each time by propagating labels from the labeled set and extra unlabeled set.

We use  miniImageNet and tieredImageNet with the labeled/unlabeled data split proposed by Ren et al. (2018). Specifically, they split the images of each class into disjoint labeled and unlabeled sets. For miniImageNet, the ratio of labeled/unlabeled data is 40% and 60% in each class. Likewise, the ratio is 10% and 90% for tieredImageNet. All semi-supervised methods (including TPN-semi) sample support/query data from the labeled set (e.g, 40% from miniImageNet) and sample unlabeled data from the unlabeled sets (e.g, 60% from miniImageNet). In addition, there is a more challenging situation where many unlabelled examples from other distractor classes (different from labelled classes).

Following Ren et al. (2018)

, we report the average accuracy over 10 random labeled/unlabeled splits and the uncertainty computed in standard error. Results are shown in Table 

3 and Table 4. It can be seen that TPN-semi outperforms all other algorithms with a large margin, especially for 1-shot case. Although TPN is originally designed to perform transductive inference, we show that it can be successfully adapted to semi-supervised learning tasks with little modification. In certain cases where we can not get all test data, the TPN-semi can be used as an effective alternative algorithm.

5 Conclusion

In this work, we proposed the transductive setting for few-shot learning. Our proposed approach, namely Transductive Propagation Network (TPN), utilizes the entire test set for transductive inference. Specifically, our approach is composed of four steps: feature embedding, graph construction, label propagation, and loss computation. Graph construction is a key step that produces example-wise parameters to exploit the manifold structure in each episode. In our method, all parameters are learned end-to-end using cross-entropy loss with respect to the ground truth labels and the prediction scores in the query set. We obtained the state-of-the-art results on miniImageNet and tieredImageNet. Also, the semi-supervised adaptation of our algorithm achieved higher results than other semi-supervised methods. In future work, we are going to explore the episodic-wise distance metric rather than only using example-wise parameters for the Euclidean distance.

Acknowledgments

Saehoon Kim, Minseop Park, and Eunho Yang were supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT1702-15. Yanbin Liu and Yi Yang are in part supported by AWS Cloud Credits for Research.

References

  • Chung and Graham [1997] Fan RK Chung and Fan Chung Graham. Spectral graph theory. American Mathematical Soc., 1997.
  • Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In

    International Conference on Machine Learning

    , pages 1126–1135, 2017.
  • Fu et al. [2015] Yanwei Fu, Timothy M Hospedales, Tao Xiang, and Shaogang Gong. Transductive multi-view zero-shot learning. IEEE transactions on pattern analysis and machine intelligence, 37(11):2332–2345, 2015.
  • Fujiwara and Irie [2014] Yasuhiro Fujiwara and Go Irie. Efficient label propagation. In International Conference on Machine Learning, pages 784–792, 2014.
  • He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, pages 770–778, 2016.
  • Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456, 2015.
  • Jia et al. [2014] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM International Conference on Multimedia, pages 675–678. ACM, 2014.
  • Joachims [1999] Thorsten Joachims. Transductive inference for text classification using support vector machines. In International Conference on Machine Learning, volume 99, pages 200–209, 1999.
  • Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), volume 5, 2015.
  • Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012.
  • Lake et al. [2011] Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Conference of the Cognitive Science Society, volume 33, 2011.
  • Lee and Choi [2018] Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pages 2933–2942, 2018.
  • Liang and Li [2018] De-Ming Liang and Yu-Feng Li. Lightweight label propagation for large-scale network data. In IJCAI, pages 3421–3427, 2018.
  • Matthias et al. [2017] Bauer, Matthias, Rojas-Carulla Mateo, Jakub Bartłomiej Świątkowski, Bernhard Schölkopf, and Richard E Turner. Discriminative k-shot learning using probabilistic models. arXiv preprint arXiv:1706.00326, 2017.
  • Mishra et al. [2018] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018.
  • Munkhdalai et al. [2018] Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In International Conference on Machine Learning, pages 3661–3670, 2018.
  • Nichol et al. [2018] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.
  • Oreshkin et al. [2018] Boris N Oreshkin, Alexandre Lacoste, and Pau Rodriguez. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, 2018.
  • Ravi and Larochelle [2017] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. International Conference on Learning Representations, 2017.
  • Ren et al. [2018] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. International Conference on Learning Representations, 2018.
  • Rohrbach et al. [2013] Marcus Rohrbach, Sandra Ebert, and Bernt Schiele. Transfer learning in a transductive setting. In Advances in Neural Information Processing Systems, pages 46–54, 2013.
  • Schmidhuber [1987] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook. PhD thesis, Technische Universität München, 1987.
  • Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
  • Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4080–4090, 2017.
  • Sugiyama [2007] Masashi Sugiyama. Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. Journal of Machine Learning Research, 8:1027–1061, 2007.
  • Sung et al. [2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Computer Vision and Pattern Recognition, 2018.
  • Thrun and Pratt [2012] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
  • Vapnik [1999] Vladimir Naumovich Vapnik.

    An overview of statistical learning theory.

    IEEE transactions on neural networks, 10(5):988–999, 1999.
  • Vinyals et al. [2016] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630–3638, 2016.
  • Wang and Zhang [2006] Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. In International Conference on Machine Learning, pages 985–992. ACM, 2006.
  • Wang et al. [2018] Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Computer Vision and Pattern Recognition, 2018.
  • Xu et al. [2017] Zhongwen Xu, Linchao Zhu, and Yi Yang. Few-shot object recognition from machine-labeled web images. In Computer Vision and Pattern Recognition, 2017.
  • Yang et al. [2016] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning, pages 40–48, 2016.
  • Zelnik-Manor and Perona [2004] Lihi Zelnik-Manor and Pietro Perona.

    Self-tuning spectral clustering.

    In Advances in Neural Information Processing Systems, 2004.
  • Zhou et al. [2004] Denny Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In Advances in Neural Information Processing Systems, pages 321–328, 2004.
  • Zhu and Ghahramani [2002] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Tech. Rep., Technical Report CMU-CALD-02–107, Carnegie Mellon University, 2002.

Appendix A Ablation Study

In this section, we performed several ablation studies with respect to training shots and query number.

a.1 Training Shots

Figure 5: Model performance with different training shots. The -axis indicates the number of shots in training, and the -axis indicates 5-way test accuracy for 1-shot and 5-shot. Error bars indicate 95% confidence intervals as computed over 600 test episodes.

a.2 Query Number

miniImageNet 1-shot
5 10 15 20 25 30
Train=15 52.29 52.95 53.75 53.92 54.57 54.47
Test=15 53.53 53.72 53.75 52.79 52.84 52.47
Train=Test 51.94 53.47 53.75 54.00 53.59 53.32
miniImageNet 5-shot
5 10 15 20 25 30
Train=15 66.97 69.30 69.43 69.92 70.54 70.36
Test=15 68.50 68.85 69.43 69.26 69.12 68.89
Train=Test 67.55 69.22 69.43 69.85 70.11 69.94
Table 5: Accuracy with various query numbers

At first, we designed three experiments to study the influence of the query number in both training and test phase: (1) fix training query to 15; (2) fix test query to 15; (3) training query equals test query. The results are shown in Table 5. Some conclusions can be drawn from this experiment: (1) When training query is fixed, increasing the test query will lead to the performance gain. Moreover, even a small test query (e.g., 5) can yield good performance; (2) When test query is fixed, the performance is relatively stable with various training query numbers; (3) If the query number of training matches test, the performance can also be improved with increasing number.

a.3 Results on ResNet

In this paper, we use a 4-layer neural network structure as described in Section 3.2.1 to make a fair comparison. Currently, there are two common network architectures in few-shot learning: 4-layer ConvNets (e.g., Finn et al. (2017); Snell et al. (2017); Sung et al. (2018)) and 12-layer ResNet (e.g., Mishra et al. (2018); Munkhdalai et al. (2018); Matthias et al. (2017); Oreshkin et al. (2018)). Our method belongs to the first one, which contains much fewer layers than the ResNet setting. Thus, it is more reasonable to compare algorithms such as TADAM (Oreshkin et al., 2018) with ResNet version of our method. To make this comparison, we implemented our algorithm with ResNet architecture on miniImagenet dataset and show the results in Table 6.

Method 1-shot 5-shot
SNAIL (Mishra et al., 2018) 55.71 68.88
adaResNet (Munkhdalai et al., 2018) 56.88 71.94
Discriminative k-shot (Matthias et al., 2017) 56.30 73.90
TADAM (Oreshkin et al., 2018) 58.50 76.70
TPN 59.46 75.65
Table 6: ResNet results on miniImageNet

It can be seen that we beat TADAM for 1-shot setting. For 5-shot, we outperform all other recent high-performance methods except for TADAM.

a.4 Closed-form solution vs Iterative updates

There is a potential concern that the closed-form solution of label propagation can not scale to large-scale matrix. We relieve this concern from two aspects. On one hand, the few-shot learning problem assumes that training examples in each class is quite small (only 1 or 5). In this situation, Eq 3 and the closed-form version can be efficiently solved, since the dimension of is only (5-way, 1-shot, 15-query) or (5-way, 5-shot, 15-query). On the other hand, there are plenty of prior work on the scalability and efficiency of label propagation, such as Liang and Li (2018); Fujiwara and Irie (2014), which can extend our work to large-scale data.

Furthermore, on miniImagenet, we performed iterative optimization and got 53.05/68.75 for 1-shot/5-shot experiments with only 10 steps. This is slightly worse than closed-form version (53.75/69.43). We attribute this slightly worse accuracy to the inaccurate computation and unstable gradients caused by multiple step iterations.

a.5 Accuracy with 95% confidence intervals

5-way Acc 10-way Acc
Model Transduction 1-shot 5-shot 1-shot 5-shot
MAML BN 48.701.84 63.110.92 31.271.15 46.921.25
MAML+Transduction Yes 50.831.85 66.191.85 31.830.45 48.231.28
Reptile No 47.070.26 62.740.37 31.100.28 44.660.30
Reptile + BN BN 49.970.32 65.990.58 32.000.27 47.600.32
PROTO NET No 46.140.77 65.770.70 32.880.47 49.290.42
PROTO NET (Higher Way) No 49.420.78 68.200.66 34.610.46 50.090.44
RELATION NET BN 51.380.82 67.070.69 34.860.48 47.940.42
Label Propagation Yes 52.310.85 68.180.67 35.230.51 51.240.43
TPN Yes 53.750.86 69.430.67 36.620.50 52.320.44
TPN (Higher Shot) Yes 55.510.86 69.860.65 38.440.49 52.770.45
  • * “Higher Way” means using more classes in training episodes. “Higher Shot” means using more shots in training episodes. “BN” means information is shared among test examples using batch normalization.

Table 7: Few-shot classification accuracies on miniImageNet. All results are averaged over 600 test episodes and are reported with 95% confidence intervals. Top results are highlighted.
5-way Acc 10-way Acc
Model Transduction 1-shot 5-shot 1-shot 5-shot
MAML BN 51.671.81 70.301.75 34.441.19 53.321.33
MAML + Transduction Yes 53.231.85 70.831.78 34.781.18 54.671.26
Reptile No 48.970.21 66.470.21 33.670.28 48.040.30
Reptile + BN BN 52.360.23 71.030.22 35.320.28 51.980.32
PROTO NET No 48.580.87 69.570.75 37.350.56 57.830.55
PROTO NET (Higher Way) No 53.310.89 72.690.74 38.620.57 58.320.55
RELATION NET BN 54.480.93 71.310.78 36.320.62 58.050.59
Label Propagation Yes 55.230.96 70.430.76 39.390.60 57.890.55
TPN Yes 57.530.96 72.850.74 40.930.61 59.170.52
TPN (Higher Shot) Yes 59.910.94 73.300.75 44.800.62 59.440.51
  • * “Higher Way” means using more classes in training episodes. “Higher Shot” means using more shots in training episodes. “BN” means information is shared among test examples using batch normalization.

Table 8: Few-shot classification accuracies on tieredImageNet. All results are averaged over 600 test episodes and are reported with 95% confidence intervals. Top results are highlighted.
Model 1-shot 5-shot 1-shot w/D 5-shot w/D
Soft -Means 50.090.45 64.590.28 48.700.32 63.550.28
Soft -Means+Cluster 49.030.24 63.080.18 48.860.32 61.270.24
Masked Soft -Means 50.410.31 64.390.24 49.040.31 62.960.14
TPN-semi 52.780.27 66.420.21 50.430.84 64.950.73
  • * “w/D” means with distraction. In this setting, many of the unlabelled data are from the so-called distraction classes , which is different from the classes of labelled data.

Table 9: Semi-supervised comparison on miniImageNet.
Model 1-shot 5-shot 1-shot w/D 5-shot w/D
Soft -Means 51.520.36 70.250.31 49.880.52 68.320.22
Soft -Means+Cluster 51.850.25 69.420.17 51.360.31 67.560.10
Masked Soft -Means 52.390.44 69.880.20 51.380.38 69.080.25
TPN-semi 55.740.29 71.010.23 53.450.93 69.930.80
  • * “w/D” means with distraction. In this setting, many of the unlabelled data are from the so-called distraction classes , which is different from the classes of labelled data.

Table 10: Semi-supervised comparison on tieredImageNet.