Few-Shot and Continual Learning with Attentive Independent Mechanisms

1 Introduction

Humans have the ability to learn new concepts continually while retaining previously learned concepts [11]. While learning new concepts, prior concepts that were learned are leveraged to form new connections in the brain [4, 52]

. The plasticity of the human brain plays an important role on the forming of novel neuronal connections for learning new concepts. Current deep learning methods are inefficient in remembering old concepts after being fed with new concepts, also widely know as catastrophic forgetting

[34, 23]. Deep neural networks (DNNs) trained in an end-to-end fashion also has difficulty in learning new tasks in a sample efficient manner [12]. It is conjectured that the cause of catastrophic forgetting and inefficiency in learning new tasks is from the stability-plasticity dilemma [2]. Stability is required so that previously learned information can be retained through the limitation of abrupt weight changes. Plasticity on the other hand encourages large weight changes, resulting in the fast acquisition of new concepts with the trade-off of forgetting old concepts.

It is believed that by scaling up the currently available architecture, DNNs are able to generalize better [7, 41, 10]. Tremendous effort is placed into neural architecture search (NAS) [28, 54, 49, 39, 32] with the hypothesis that improvements on a structural level introduce inductive bias that improves the generalizability of a neural network. As most of the prior arts are evaluated on benchmark datasets that are distributed similarly to the training set that it is trained on, the evaluation results are not a good measure of the generalization. We argue that the ability to adapt, acquire new knowledge and recall previously learned information plays an important role in reaching true generalization. The importance of learning to learn, i.e. meta-learning, has shone the spotlight on two major research direction that we will focus on — few-shot learning and continual learning. In few-shot learning [12, 37, 45, 14], the goal is to learn novel concepts with as few samples as possible, i.e. evaluating the capability of adapting to new tasks. Whereas in continual learning, the ability to learn an increasing amount of concepts while not forgetting old ones is evaluated.

Following OML [22], we separate the feature extraction part and the decision making part of the network, defined in OML as representation learning network (RLN) and prediction learning network (PLN) respectively. The fast and slow learning in OML is performed on an architecture level, i.e. RLN is updated in the outer loop (slow weights) and PLN is updated in the inner loop (fast weights). Such approach has proven to be helpful in learning sparse representation that are beneficial for fast adaptation and prevention of catastrophic forgetting. We take one step further by introducing sparsity on an architectural level, accomplished through the introduction of Attentive Independent Mechanisms (AIM). AIM is composed of a set of mechanisms that competitively attend to the input representation, having mechanisms that are closely related to the input representation being activated during inference. AIM can be understood as a mixture of experts competing to explain an incoming representation, hence only the mechanisms that best explain the input representation will be updated, leading to a sparse representation or modeling on an architectural level. Having sparse modeling on an architectural level for higher-order representations has its benefits, as only the experts or mechanisms that best explain a task will be involved in the learning process, helping in the acceleration of learning new concepts and the mitigation of catastrophic forgetting. To demonstrate the potential of AIM as a fundamental building block for fast learning without forgetting, we demonstrate its strength on few-shot classification [12, 43, 53] and continual learning [5, 22, 23] benchmarks.

Our contributions are as follows: (1) In Section 3, we give a detailed description and formulation of AIM — a novel module that can be used for few-shot and continual learning. (2) We apply AIM on few-shot learning and continual learning tasks in Section 4.1 and Section 4.2 respectively. Qualitative and quantitative results are shown for both learning tasks, giving readers an insight on the importance of having AIM in the context of few-shot and continual learning. For few-shot classification, experiments are performed on CIFAR-FS and MiniImageNet whereas for continual learning, experiments are performed on Omniglot, CIFAR-100 and MiniImageNet. Substantial improvement in accuracy over prior arts are shown.

2 Related Work

Meta-learning revolves on the idea of learning to learn, hoping that through the observation of training iterations on a few tasks, we are able to generalize to unseen tasks with only a few or zero samples. Meta-learning is usually composed of a support set and a query set. The support set is used for fast adaptation and the query set is used to evaluate the adapted model and to meta-learn the adaptation procedure. Model-based meta-learning methods include the work by [35] that uses a meta-learner based on a LSTM [18] which includes all previously seen samples, i.e. all support samples of a task are considered during the class prediction of query samples through an attentive mechanism. Another similar work by [44] augments LSTM with an external memory bank. [36] incorporates fast and slow weights for few-shot classification.

Metric-based meta-learning methods include Siamese Network proposed by [24] which predicts whether two images originate from the same class. [50] proposed Matching Networks that uses cosine distance in an attention kernel to measure the similarity of images in its embedding space. [45] later found that using Euclidean distance as a metric instead of cosine distance improves performance. A generalization of all the mentioned work is done by modeling the metric using a graph neural network proposed by [13].

Optimization-based meta-learning includes [42] that proposed using a LSTM meta-learner which provides gradient to a convolutional network-based fast learner. [12, 37]

proposed an inner and outer-loop optimization method having fast adaptation in the inner-loop and an outer-loop update that backpropagates through the inner-loop updates.

[53] used the concept of inner and outer loop-update by having the context parameters (embeddings of tasks) updated in the inner-loop. LEO [43]

has its classifier weights generated by a low-dimensional latent embedding updated in the inner-loop.

[15]

proposed a similar approach where classification weights are generated using feature vectors that corresponds to the support set. SIB

[20] performs transductive inference using synthetic gradient [21] on the feature averaging variant classifier proposed by [15]. Transductive inference was first introduced to the context of few-shot classification by [33], having a graph constructed for the support set and the query set, with labels propagated within the graph. As the architecture proposed by [33] is restrictive, [19] proposed a more general approach that uses a cross attention module that models semantic relevance between the support and query set.

In continual learning, the objective is to mitigate catastrophic forgetting [23]. Earlier works are based on regularization method, with [17] proposing the use of fast and slow training weights, borrowing the idea of plasticity and stability for network training. This idea is then adopted by OML [22] to learn representations that are useful for future learning and helps in mitigating catastrophic forgetting. Similarly, fast and slow learning is applied to ANML [5], having a neuromodulatory network modeled using slow weights. [1] uses task-specific gate module and prediction head to reduce competitive effect between classes. A criterion is designed in [3] to store most-interfered samples in a fixed-sized rehearsal memory.

3 Method

Figure 1: AIM is inserted right after the feature extractor $f_{ψ}$ and before the output classifier $ϕ$ . Only mechanisms closely related to the input representation are active (green boxes) and updated during the training phase (blue dashed lines).

As Attentive Independent Mechanisms (AIM) is used to model higher order information, we place it right after a feature extractor, defined as $z = f_{ψ} (x)$ . $f_{ψ} (\cdot)$ is a series of convolutional layers parameterized by $ψ$ , $x$ is an input sample and $z$ is its corresponding representation. AIM is a module that is parameterized by $W$ and is defined as $A_{W}$ . The representation from AIM is then fed to a linear layer $ϕ$ for the task of classification. An illustration of AIM as a module is shown in Figure 1. We also show an illustration on the application of AIM to existing meta-learning frameworks used for few-shot learning and continual learning in Figure 2. We first describe the implementation of AIM as module in Section 3.1 followed by its integration to SIB [20] for few-shot learning in Section 3.2 and to OML [22] and ANML [5] for continual learning in Section 3.3.

3.1 Attentive Independent Mechanisms

The goal of AIM is to learn a sparse set of mechanisms, i.e. mixture of experts, to decouple the modeling of higher order information from the feature extraction pipeline. These mechanisms compete and attend to the input representation in a top-down fashion using cross-attention [30, 29]. Through the strict selection of mechanisms, a sparse set of mechanisms will be selected for every task, inducing an architectural bias that helps in fast adaptation to new tasks and mitigating catastrophic forgetting. The structure of AIM is composed of a set of independent mechanisms, each parameterized by its own set of parameters. Each mechanism acts as an independent expert that collaborate with other experts to solve a particular task. AIM can be viewed as a static version of RIMs [16], i.e. temporal modeling of hidden states using LSTM [18] found in RIMs is removed. For RIMs, the model is fed with a continuous stream of inputs, making dynamical modeling using LSTM intuitive. For AIM, the assumption of having continuous stream of inputs does not hold as the practice of few-shot classification and continual learning have i.i.d. data being fed into the model during training and inference. Departing from RIMs, the objective of AIM is to show that through a mixture of experts, new concepts can be easily learned with minimal catastrophic forgetting. We hypothesize that by having a set of independent mechanisms, a sparse set of factorized representations or concepts can be extracted from the input representation. Such concepts have properties that are tasks-invariant which can be helpful in learning new tasks. The learning of concepts in AIM can also be understood as the amortized version of memory based models that stores samples either in the form of images or representations [44], which scales with the size of tasks in the system without limitation. AIM on the other hand performs implicit modeling of samples, analogous to the amortized modeling using a DNN instead of using a non-parametric method that stores samples from the training set for inference [8].

Following RIMs, AIM has a null vector $\emptyset$ that is concatenated with the input representation $z$ , giving us $^z = [z^{T}, \emptyset^{T}]^{T}$ . The mechanisms then attend to the incoming latent representation $^z$ as:

~ z =^z (M \sum m = 1 w_{m} (^z) W_{m}^{M}),

(1)

which could be understood as the passing of input representation $^z$ through the weighted-summation of the mechanism weights, $W_{m}^{M}$ . The summation of the outputs of the mechanisms makes the extension to arbitrary number of mechanisms trivial when compared to the concatenation of outputs used in RIMs. Concatenation is also infeasible when the output dimension of $W_{m}^{M}$ is large, resulting in a wide input dimension for the upcoming layer. The summation of mechanisms also has the property of permutation invariance, reducing the complexity of the output classifier $ϕ$ .

To encourage sparsity, we enforce the mechanisms to compete with each other to attend to the incoming representation. This is done by having only the weights of mechanisms that are closely related to the input representation to be selected, i.e. only top $K$ mechanisms out of a total of $M$ mechanisms are selected for the downstream prediction tasks. The strict selection of mechanisms forces the mechanisms to compete with each other to attend to the incoming signal, modeling the biased competition theory of selective attention [9]. The selection of mechanisms is given as:

wm(^z)={˜wm(^z),if m∈topK(˜w1(^z),…,˜wM(^z)),0,otherwise.

(2)

The indices corresponding to the top $K$ values from a set is returned by the ${t o p}_{K} (\cdot)$ operator. The weights used to weight the importance of the selected mechanisms are composed of the softmax of the normalized inner-product, $⟨ \cdot, \cdot ⟩$ , between the mechanisms’ hidden state $h_{m}$ and the input representation $z$ that are first mapped to a lower-dimensional embedding by the query weight $W_{m}^{Q}$ and key weight $W^{K}$ of output dimension $d$ respectively, given as:

(3)

Note that $s o f t m a x$ is applied locally for each mechanism, i.e. the transformation of the attention values corresponding to $z$ and $\emptyset$ into a probabilistic one. The value that corresponds to the input (not null) dimension from (10) is used for the top $K$ comparison in (9).

Intervention during training.

The training of AIM can be understood as an intervention procedure with the model selecting a few mechanisms to be included during the forward pass phase of training. Mechanisms that perform well on the training data are rewarded by having gradient update directed to the activated mechanisms, with the sensitivity to novel inputs reflected on $h_{m}$ . As one can predict, there is a possibility of the occurrence of mechanism-overfitting, where only a fixed set of mechanisms are active for all training tasks, losing the original motivation of having a sparse set of mechanisms acting as experts on different tasks. Mechanism-overfitting is also equivalent to having a DNN with multiple residual paths, resembling a single layer of Inception [48], diverting from our original goal of building models that are invariant across tasks.

To prevent the collapsing towards having only a few active mechanisms for all tasks, the trick is to enable the exploration of different amount of mechanisms during training, instead of locking down to the top $K$ mechanisms. Stochasticity is introduced into the selection process by sampling top $K + l$ (also known as stochastic sampling count) instead of top $K$ mechanisms. We then perform uniform sampling without replacement of $K$ mechanisms from the top $K + l$ mechanisms, where the original sampling condition of (9) can now be written as:

wm(^z)=⎧⎪⎨⎪⎩˜wm(^z),if m∈{K⊆S∣|K|=K},s.t. S=topK+l(˜w1(^z),…,˜wM(^z))0,otherwise.

(4)

Here, $| \cdot |$ is the cardinality operator to ensure that the sampled subset $K$ is of size $K$ and is sampled without replacement. Such intervention is analogous to stochastic intervention [25] and dropout [46] which adds stochasticity to the training of AIM, preventing the locking down to a few mechanisms that are attended to upon initialization.

Training and evaluation of AIM.

Weight updates in AIM is similar to a typical layer in DNNs, i.e. gradients are backpropagated from the final loss function. A distinct difference from a conventional module in DNNs is that only the mechanisms activated during a forward pass are updated, resulting in a sparse set of weight updates. As AIM is designed to model higher order concepts, it is placed in the higher level of a DNN and has fast weights that are updated in the inner-loop of a meta-learning pipeline. The role of AIM as a module is shown in Figure

1. The procedure for the meta-training of AIM for both few-shot learning and continual learning is shown in Algorithm 1, whereas the meta-testing counterpart is shown in Algorithm 2. The algorithms shown are applicable for both few-shot and continual learning with the distinction between both highlighted with different colors — few-shot learning using SIB in green and continual learning using OML and ANML in blue. Step sizes for the inner-loop and outer-loop are defined as

ν^{in}

and

ν^{out}

respectively. Step size for synthetic gradient update used for SIB is defined as

ϵ

. For few-shot learning, the fast adaptation of AIM is evaluated using the meta-testing test set of the sampled task, i.e.

S_{test}

in the outer-loop. For continual learning, evaluation is performed after the completion of meta-training, and is tested on the entire meta-test train set

S_{train}^{'}

and meta-test test set

S_{test}^{'}

N

sequential tasks

T

; step size

ν^{i n}, ν^{o u t}, ϵ

; inner iterations

T

; modules

f_{ψ}, A_{W}, ϕ, θ

(SIB only)

2:while not done do

{S_{train}, S_{test}} \sim T

▹

SIB: i.i.d.; continual: sequential

4: for

t \leftarrow 1, T

5: Update fast weights using

S_{train}

▹

step size:

ν^{i n}

6: SIB:

A_{W}

OML:

A_{W}, ϕ

ANML:

f_{ψ^{P}}, A_{W}, ϕ

7: end for

8: Update

ϕ

using transductive inference

▹

step size:

ϵ

9: Update slow weights using

S_{test}

▹

step size:

ν^{o u t}

10:

SIB:

θ

OML:

f_{ψ}

ANML:

f_{ψ^{N M}}

11:end while

Algorithm 1 Meta-Training: Training of AIM

N

sequential unseen tasks

T

; step size

ν^{i n}, ϵ

; inner iterations

T

; modules

f_{ψ}, A_{W}, ϕ, θ

(SIB only)

S_{train}^{'} = {}

;

S_{test}^{'} = {}

▹

initialize empty set

3:for

n \leftarrow 1, N

{S_{train}, S_{test}} \sim T_{n}

▹

SIB: i.i.d.; continual: sequential

S_{train}^{'}, S_{test}^{'} = {S_{train}^{'}, S_{train}}, {S_{test}^{'}, S_{test}}

▹

store trajectory

6: for

t \leftarrow 1, T

7: Update fast weights using

S_{train}

▹

step size:

ν^{i n}

8: SIB:

A_{W}

OML:

A_{W}, ϕ

ANML:

f_{ψ^{P}}, A_{W}, ϕ

9: end for

10: Update

ϕ

using transductive inference

▹

step size:

ϵ

11: Evaluate on

S_{test}

12:end for

13:Evaluate on

S_{train}^{'}

▹

end of meta-test training trajectory

14:Evaluate on

S_{test}^{'}

▹

eval on entire meta-test testing set

Algorithm 2 Meta-Testing: Evaluation of AIM

(a) Synthetic Information Bottleneck (SIB) [20]

3.2 Few-Shot Learning Using SIB

SIB is composed of two works: synthetic gradient modeling [21] and a feature averaging classifier [15]. In [21], the idea is to use a synthetic gradient model, $S$ , that is meta-learned to generate gradient when labeled data is absent for transductive inference, i.e. update of weights without gradients propagated from a loss that is dependent on label. In [15]

, a classifier is defined as the cosine similarity between feature representations

~ z

and classification weight vectors

ϕ

ϕ

is generated using an external classification weight generator

G_{θ} (\cdot)

parameterized by

θ

followed by iterative update by the synthetic gradient model

S

. Feature vectors of

P

training samples of a novel category

¯ Z = {{¯ z}^{(i)}}_{i = 1}^{P}

are fed as input to generate a new set of weights for classification,

ϕ^{'} = G_{θ} (¯ Z)

. In SIB, feature averaging based weight inference is used, i.e. the classification weight vector is obtained as

ϕ^{'} = θ ⊙ w_{a v g}

, where

⊙

is the Hadamard product and

w_{a v g} = \frac{1}{P} \sum_{i = 1}^{P} {¯ z}^{(i)}

(

¯ z

is the

ℓ_{2}

-normalized version of

~ z

). The classification weight vector

ϕ^{'}

is then updated iteratively using the synthetic gradient model in SIB, given as

ϕ = S (ϕ^{'})

. Both the synthetic gradient model and the weights of the weight generator

θ

are meta-learned, i.e. updated in the outer-loop. To encourage sparse modeling of higher order concepts in the network, AIM is inserted right after the feature extractor

f_{ψ} (\cdot)

and before the output linear classifier

ϕ

that is generated using

G_{θ} (¯ Z)

and

S

, or,

(5)

Training.

Following the training pipeline in SIB [20], the weights of the feature extractor $ψ$ are frozen to simplify the training procedure. The weights of the AIM, $W$ , and the output linear classifier, $ϕ$ , are updated as fast weights, i.e. inner-loop. Only the weights of the classification weight generator $θ$ are updated as slow weight, i.e. outer-loop. The application of AIM to SIB is shown in Figure (a)a.

3.3 Continual Learning: Learning Fast and Slow

It is shown in the task of continual learning that learning fast and slow from the context of meta-learning is helpful for the mitigation of catastrophic forgetting [22, 5]. OML [22] and ANML [5] are example frameworks for continual learning that uses this methodology, showing promising results. To validate our claim on the importance of incorporating sparse modeling on an architectural level for the mitigation of catastrophic forgetting, we insert AIM into both OML and ANML and observe the resulting performance.

Oml.

The entire architecture is split into two parts — representation learning network (RLN) and prediction learning network (PLN). RLN uses slow weights and PLN uses fast weights. Following our notations, RLN is the feature extractor in our work, $f_{ψ}$ , and PLN is the classifier (not limited to a single layer), $ϕ$ , in our work. AIM is inserted after the RLN and before the PLN, or,

y = ϕ (A_{W} (f_{ψ} (x))) .

(6)

AIM is trained jointly with PLN, i.e. they have fast weights. The application of AIM to OML is shown in Figure (b)b.

Anml.

Two set of feature extractors are used in ANML — a neuromodulatory network, $f_{ψ^{N M}}$ , and a prediction network, $ϕ \cdot f_{ψ^{P}}$ . The role of the neuromodulatory network is to modulate the latent representation of the prediction network, i.e. the output of $f_{ψ^{P}}$ in Figure (c)c. The output of the neuromodulatory network is element-wise multiplied with the outout of $f_{ψ^{P}}$ before passing to the final classifier, or $y = ϕ (f_{ψ^{N M}} (x) ⊙ f_{ψ^{P}} (x))$ . Only the neuromodulatory network has slow weights and the entire prediction network has fast weights. Similar to SIB and OML, AIM is inserted right after the feature extractor and uses fast weights,

y = ϕ (A_{W} (f_{ψ^{N M}} (x) ⊙ f_{ψ^{P}} (x))) .

(7)

4 Experiments

4.1 Few-Shot Learning

Datasets.

For all datasets, class splits are disjoint. MiniImageNet [50] contains a total of 100 classes which are split into 64 training, 16 validation and 20 testing classes; images are of size $84 \times 84$ . CIFAR-FS [6] is created by dividing CIFAR-100 into 64 training, 16 validation and 20 testing classes; images are of size $32 \times 32$ . For few-shot classification, each task (episode) consists of a train set and a test set. For each task, $k$ classes are sampled from the class pool mentioned. For each class, $n$ examples are drawn and are relabeled as $k$ disjoint classes forming the train set. For the test set, 15 $k$ samples are used. We show results of $k = 5$ for both $n = 1$ and $n = 5$ .

Network architecture.

We follow the setup in [20, 15, 40, 14] by using a 4-layer convolutional network with 64 feature channels (Conv-4-64) or a WideResNet (WRN-28-10) [51] as our feature extractor, $f_{ψ}$ . $f_{ψ}$

is pretrained in a typical end-to-end supervised learning fashion, i.e. the entire training set is used for batch update. Our classifier is adopted directly from

[20, 15] having

ϕ = G_{θ}^{'} (¯ Z)

. For transductive inference [20], the synthetic gradient network is modeled by a MLP of 3 layers and hidden size 8

k

. Classification is done by using the cosine-similarity based classifier found in [20, 15]. For AIM, all weights are linear layers. The hidden state

h_{m}

of the mechanisms are of dimension 256. The key and query weights (

W^{K}

and

W_{m}^{Q}

) maps the input and hidden state to a dimension of 128 to perform distance measurement. For the output dimension of the mechanism weights,

W_{m}^{M}

, we picked 400 for CIFAR-FS trained on Conv-4-64 and 800 for the rest; this decision is based on the dimension of the flattened feature map at the output of the feature extractor (not cherry-picked).

Training details.

We use $M = 32$ mechanisms with top $K = 8$ mechanisms selected during inference with induced stochasticity by having $l = 2$ during training. SGD is used a batch size of 1 for 50,000 steps with learning rate $ϵ = 10^{- 3}$ for SIB’s classifier synthetic update, $ν^{o u t} = 5 \times 10^{- 3}$ for outer-loop update and $ν^{i n} = 3 \times 10^{- 3}$

for inner-loop update. The feature extractor is frozen during training. 1,000 tasks are sampled from the validation set for hyperparameter selection at each training epoch. All experiments are run on a single GTX1080Ti using PyTorch. A complete run of Conv-4-64 on CIFAR-FS and WRN-28-10 on MiniImageNet takes less than 2 hours and 5 hours respectively.

(a) SIB + AIM using Conv-4-64 on CIFAR-FS, 1-shot.

Method	Backbone	Transductive	MiniImageNet, 5-Way		CIFAR-FS, 5-Way
Method	Backbone	Transductive	1-shot	5-shot	1-shot	5-shot
Matching Net [50]	Conv-4-64		44.2%	57%	-	-
MAML [12]	Conv-4-64		$48.7 \pm 1.8 %$	$63.1 \pm 0.9 %$	$58.9 \pm 1.9 %$	$71.5 \pm 1.0 %$
Prototypical Net [45]	Conv-4-64		$49.4 \pm 0.8 %$	$68.2 \pm 0.7 %$	$55.5 \pm 0.7 %$	$72.0 \pm 0.6 %$
Relation Net [47]	Conv-4-64		$50.4 \pm 0.8 %$	$65.3 \pm 0.7 %$	$55.0 \pm 1.0 %$	$69.3 \pm 0.8 %$
TPN [33]	Conv-4-64	✓	55.5%	69.9%	-	-
Gidaris [14]	Conv-4-64		$54.8 \pm 0.4 %$	$71.9 \pm 0.3 %$	$63.5 \pm 0.3 %$	$79.8 \pm 0.2 %$
SIB [20]	Conv-4-64	✓	$58.0 \pm 0.6 %$	$70.7 \pm 0.4 %$	$68.7 \pm 0.6 %$	$77.7 \pm 0.4 %$
SIB + Linear layer	Conv-4-64	✓	$60.07 \pm 0.59 %$	$73.70 \pm 0.38 %$	$68.75 \pm 0.62 %$	$79.99 \pm 0.39 %$
AIM (ours)	Conv-4-64	✓	$61.90 \pm 0.57 %$	$74.55 \pm 0.38 %$	$71.09 \pm 0.62 %$	$80.48 \pm 0.40 %$
TADAM [38]	ResNet-12		$58.5 \pm 0.3 %$	$76.7 \pm 0.3 %$	-	-
SNAIL [44]	ResNet-12		$55.7 \pm 1.0 %$	$68.9 \pm 0.9 %$	-	-
CTM [31]	ResNet-18	✓	$64.1 \pm 0.8 %$	$80.5 \pm 0.1 %$	-	-
LEO [43]	WRN-28-10		$61.8 \pm 0.1 %$	$77.6 \pm 0.1 %$	-	-
Gidaris [14]	WRN-28-10		$62.9 \pm 0.5 %$	$79.9 \pm 0.3 %$	$73.6 \pm 0.3 %$	$86.1 \pm 0.2 %$
SIB [20]	WRN-28-10	✓	$70.0 \pm 0.6 %$	$79.2 \pm 0.4 %$	$80.0 \pm 0.6 %$	$85.3 \pm 0.4 %$
SIB + Linear layer	WRN-28-10	✓	$67.38 \pm 0.54 %$	$80.54 \pm 0.34 %$	$78.02 \pm 0.55 %$	$79.91 \pm 0.38 %$
AIM (ours)	WRN-28-10	✓	$71.22 \pm 0.57 %$	$82.25 \pm 0.34 %$	$80.20 \pm 0.55 %$	$87.34 \pm 0.36 %$

Table 1:

Average classification accuracies with 95% confidence intervals on the test-set of MiniImageNet and CIFAR-FS. 2000 episodes are sampled for MiniImageNet and CIFAR-FS using Conv-4-64 and WRN-28-10 as the feature extractor.

4.1.1 Qualitative Study: Activation of AIM

We show heatmaps that illustrate mechanisms activated for different classes from the validation set in Figure 3. The heatmap is plotted by averaging the mechanisms’ activity for each class over the entire validation set, with 1 and 0 indicating active and inhibited mechanism respectively. We can observe that there’s a set of mechanisms that are shared among tasks and another set that are distributed sparsely. The sharing of mechanisms can be understood as different classes sharing similar concepts. The sparse allocation of mechanisms over different classes show that there are features that are invariant for certain classes only, improving resiliency to covariate shift among distributions.

4.1.2 Quantitative Study

Stochastic sampling count.

To show the importance of inducing stochasticity in the mechanism selection process for inference, we perform an empirical study by varying the stochastic sampling count, $K + l$ . We fix $K = 8$ and vary $l$ from 0 to 24. As we can see from Figure (a)a, the accuracy obtained by varying $l$ have different maximum for different datasets, models and number of shots. For most cases, the peak accuracy usually occurs at small value of $l$ and slowly deteriorates as more stochasticity is introduced.

Number of active mechanisms.

An interesting question would be how maybe active mechanisms are required to reap the benefits of sparse activations. Empirical study is performed as shown in Figure (b)b, showing the accuracy obtained by varying the number of active mechanisms $K$ from 1 to 32. The results show that accuracy is low when $K$ is small and saturates for larger values of $K$ . This shows that a limited set of active mechanisms is sufficient. Sparsity in representation can still be met when the number of active mechanisms is large, but it will be cost inefficient during both training and inference.

Benchmark evaluation.

As AIM is introduced as an additional component that’s integrated into SIB [20], the gain in accuracy shows the importance of having a mixture of experts for fast adaptation. We also show the results for SIB with a linear layer (parameters equal the total parameters found in the AIM module) added before the classifier (SIB + Linear) to show that the gain in accuracy from AIM is not solely from the increase in parameters. From Table 1, we can see that AIM outperforms all existing few-shot classification methods by a noticeable margin. As only a single layer of AIM is explored, the coupling between AIM as found in RIMs [16] is not considered here. We believe that further improvements can be attained if layers of AIM are stacked, with coupling between them considered.