Multilinear Compressive Learning with Prior Knowledge

The recently proposed Multilinear Compressive Learning (MCL) framework combines Multilinear Compressive Sensing and Machine Learning into an end-to-end system that takes into account the multidimensional structure of the signals when designing the sensing and feature synthesis components. The key idea behind MCL is the assumption of the existence of a tensor subspace which can capture the essential features from the signal for the downstream learning task. Thus, the ability to find such a discriminative tensor subspace and optimize the system to project the signals onto that data manifold plays an important role in Multilinear Compressive Learning. In this paper, we propose a novel solution to address both of the aforementioned requirements, i.e., How to find those tensor subspaces in which the signals of interest are highly separable? and How to optimize the sensing and feature synthesis components to transform the original signals to the data manifold found in the first question? In our proposal, the discovery of a high-quality data manifold is conducted by training a nonlinear compressive learning system on the inference task. Its knowledge of the data manifold of interest is then progressively transferred to the MCL components via multi-stage supervised training with the supervisory information encoding how the compressed measurements, the synthesized features, and the predictions should be like. The proposed knowledge transfer algorithm also comes with a semi-supervised adaption that enables compressive learning models to utilize unlabeled data effectively. Extensive experiments demonstrate that the proposed knowledge transfer method can effectively train MCL models to compressively sense and synthesize better features for the learning tasks with improved performances, especially when the complexity of the learning task increases.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

05/17/2019

Multilinear Compressive Learning

Compressive Learning is an emerging topic that combines signal acquisiti...
09/22/2020

Performance Indicator in Multilinear Compressive Learning

Recently, the Multilinear Compressive Learning (MCL) framework was propo...
01/15/2019

Analysis of non-stationary multicomponent signals with a focus on the Compressive Sensing approach

The characterization of multicomponent signals with a particular emphasi...
11/21/2013

Compressive Measurement Designs for Estimating Structured Signals in Structured Clutter: A Bayesian Experimental Design Approach

This work considers an estimation task in compressive sensing, where the...
12/26/2018

Uncertainty Autoencoders: Learning Compressed Representations via Variational Information Maximization

The goal of statistical compressive sensing is to efficiently acquire an...
04/28/2021

Weighed ℓ_1 on the simplex: Compressive sensing meets locality

Sparse manifold learning algorithms combine techniques in manifold learn...
09/04/2019

What Happens on the Edge, Stays on the Edge: Toward Compressive Deep Learning

Machine learning at the edge offers great benefits such as increased pri...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Compressive Sensing (CS) [11]

is an efficient signal acquisition paradigm that performs the measurement of the signal at sub-Nyquist rates by sensing and linearly interpolating the samples at the sensor level. Due to the property that the compression step is performed before signal registration during the sampling phase, CS significantly lowers the temporary storage and bandwidth requirement of the sensing devices. Therefore, this paradigm plays an important role in many applications that involve high-dimensional signals since the signal collection process can be very computationally demanding, time-consuming, or even prohibitive when using the traditional point-based sensing-then-compressing approach

[8]. For example, Hyperspectral Compressive Imaging (HCI) [3] and Synthetic Aperture Radar (SAR) imaging [34] often perform remote sensing of large volume of data constrained by limited electrical power, storage capacity and transmission bandwidth, or in 3D Magnetic Resonance Imaging (MRI) [23], long scanning time can make patients feel uncomfortable, promoting body movements, thus degrading the image quality.

Although the ideal sampling theorem requires the signal to be sampled at higher rates than the Nyquist rate to ensure perfect reconstruction, in CS, the undersampled signal can still be reconstructed almost perfectly if the sparsity assumption holds and the sensing operators possess certain properties [10, 17]. The sparsity prior assumes the existence of a set of basis or a dictionary in which the signal of interest has sparse representations, i.e., having few non-zero coefficients. Sparsity manifests in many classes of signals that we operate on since they are often smooth or piece-wise smooth, thus having sparse representations in Fourier or wavelet domains. This form of prior knowledge incorporated into the model allows us to acquire the signals at a much lower cost.

In fact, over the past decade, a large body of works in CS has been dedicated to finding relevant priors from the class of signals of interest, and incorporating them into the model to achieve better compression and reconstruction [33]. A typical form of priors that has been widely used in CS is the existence of a signal similar to the original signal of interest. This information is available in many scenarios such as video acquisitions [39, 24]

and estimation problems

[13] where signals acquired in succession are very similar. Similarity also exists in signals captured by nearby sensors in sensor networks [6] or images from spatially close cameras in multiview camera systems [12, 43]. Other examples of prior information include the available estimate of the support of the signal [44]

, Gaussian mixture models

[37], weighted sparsity [25] or block sparsity [40, 18]. The process of finding prior information often relies on the available prior knowledge related to the signal, e.g., the spatial arrangement of sensor networks, while the incorporation of such information is often expressed as another term to be minimized such as the difference between the current frame and the last frame when reconstructing MRI images. For a detailed discussion on prior information in CS and its theoretical and empirical improvements, we refer the readers to [33].

While signal reconstruction is a necessary processing step in some applications, there exist many scenarios in which the primary purpose is to detect certain patterns or to infer some properties from the acquired signal, rather than the recovery of the entire signal. Learning from compressed measurements, also known as Compressive Learning (CL) [9, 15, 14, 35, 29, 1, 41], focuses on the task of inference directly from compressive domain without explicit signal reconstruction. That is, CL models are often formed under the assumption that only certain aspects or representative features of the signal need to be captured to make inferences, even when perfect signal recovery might be impossible from the retained information. It is worth noting that in many scenarios such as security and surveillance applications [45, 32], signal recovery might disclose privacy information. Thus, it is undesirable.

Since the main objective of CL is to extract relevant information given the learning problem, the literature in CL mainly concerns with the selection of the sensing operators [36, 4] or the optimization of the learning model [15] that operates in the compressive domain. In the early works, the selection or design of sensing matrices was decoupled from the optimization of the inference model. Nowadays, with the developments on both hardware and algorithmic levels, stochastic optimization has become more and more efficient, making end-to-end learning systems applicable and appealing to a wide range of applications. Initiated by the work of [30, 1], recent CL systems [22, 16, 49, 41]

have adopted an end-to-end approach which jointly optimizes the sensing matrices and the inference operators, leaving the task of discovering relevant features to stochastic gradient descent.

The majority of CL systems employs linear sensing operators which operate on the vector-based input, regardless of the natural representation of the signal. Recently, the authors in

[41]

proposed Multilinear Compressive Learning (MCL) framework, which takes into consideration the natural structure of multidimensional signals by adopting multilinear sensing and feature extraction components. MCL has been shown both theoretically and empirically to be more efficient than its vector-based counterpart. The assumption made in MCL is that the original signals can be linearly projected along each tensor mode to a tensor subspace which retains relevant information for the learning problem, and from which discriminative features can be synthesized. In a general setting, the authors in

[41]

propose to initialize the sensing matrices with the left singular vectors obtained from Higher Order Singular Value Decomposition (HOSVD), a kind of

weak prior that initially guides the model towards an energy-preserving tensor subspace during stochastic optimization.

As we have seen from CS literature, finding good priors and successfully incorporating them into the signal model lead us to better solutions. While prior information exists in many forms in the reconstruction task, it is not straightforward to define relevant information given the learning problem. Indeed, representation learning [7]

has been an active research area that aims to extract meaningful representations from the raw data, ideally without any human given label such as object categories. From the human perception, in certain cases, we do have an idea of what kind of information is relevant or irrelevant to the learning task, e.g., color cues might be irrelevant in face recognition problems but relevant for the traffic light detection problem. From the optimization point of view, it is hard to make such a claim before any trial. Therefore, we often want to strike a balance between hand-crafted features and end-to-end solutions.

In this paper, we aim to tackle the problems of How to find good priors in Compressive Learning? and How to incorporate such priors into an end-to-end Compressive Learning system without hard-coding? Based on two observations:

  • Although the sensing operators in CL are limited to linear/multilinear form, the feature synthesis component that operates on compressed measurements can adopt nonlinear transformations.

  • Nonlinear sensing operators have better capacities to discover representative tensor subspaces

we propose a novel approach that utilizes nonlinear compressive learning models to discover the structures of the compressive domain and the informative features that should be synthesized from this domain for the learning task. This knowledge is subsequently transferred to the compressive learning models via progressive Knowledge Distillation [21]. Our contributions can be summarized as follows:

  • In this paper, a novel methodology to discover and incorporate prior knowledge into existing end-to-end Compressive Learning systems is proposed. Although we limit our investigation of the proposed method to Multilinear Compressive Learning framework, the approach described in this paper can be applied to any end-to-end Compressive Learning system to improve its performance.

  • The proposed approach naturally leads to the semi-supervised extension in which the availability of unlabeled signals coming from similar distributions can benefit Compressive Learning systems via prior knowledge incorporation.

  • We carefully designed and conducted extensive experiments in object classification and face recognition tasks to investigate the effectiveness of the proposed approach to learn and incorporate prior information. In addition, to facilitate future research and reproducibility, we publicly provide our implementation of all experiments reported in this paper111https://github.com/viebboy/MultilinearCompressiveLearningWithPrior.

The remainder of our paper is organized as follows: in Section II, we review the related works in Compressive Learning, Knowledge Distillation, and give a brief description of Multilinear Compressive Learning framework. Section III provides a detailed description of our approach to learn and incorporate prior knowledge into MCL models in both supervised and semi-supervised settings. Section IV details our empirical analysis, including experiment protocols, experiment designs, and quantitative and qualitative evaluation. Section V concludes our work.

Abbreviation & Nomenclature

[]

CS

Compressive Sensing

CL

Compressive Learning

MCL

Multilinear Compressive Learning

MCLwP

Multilinear Compressive Learning with Priors

HOSVD

Higher Order Singular Value Decomposition

KD

Knowledge Distillation

FS

Feature Synthesis

Multi-dimensional analog signal

Compressed measurements (vector) in CL

Compressed measurements (tensor) in MCL

Tensor features, output of FS component in MCL

CS component in MCL

Parameters of

FS component in MCL

Parameters of

Task-specific neural network in MCL

Parameters of

Prior-generating model

Compressed measurements in

Tensor features, output of FS component in

Nonlinear sensing component in

Parameters of

FS component in

Parameters of

Task-specific neural network in

Parameters of

-th data sample

label of the -th sample

Number of labeled samples

Total number of samples, including labeled and unlabeled samples

The set of labeled data

The set of unlabeled data

Enlarged labeled set

Inference loss function

Distillation loss function

Ii Related Work

Ii-a Background

Throughout the paper, we denote scalar values by either lower-case or upper-case characters , vectors by lower-case bold-face characters , matrices by upper-case or Greek bold-face characters and tensor as calligraphic capitals . A tensor with modes and dimension in the mode- is represented as . The entry in the -th index in mode- for is denoted as . In addition, denotes the vectorization operation that rearranges elements in to the vector representation.

The mode- product between a tensor and a matrix is another tensor of size and denoted by . The element of is defined as .

In CS, given the original analog signal , signal acquisition model is described via the following equation:

(1)

where denotes the measurements obtained from CS devices and denotes the sensing operator or sensing matrix that performs linear interpolation of .

For a multidimensional signal , we have Multilinear Compressive Sensing (MCS) model as follows:

(2)

where is the compressed measurements having a tensor form, and

denotes separable sensing operators that perform linear transformation along each mode of the original signal

.

Since Eq. (2) can also be written as:

(3)

where , , and denotes the Kronecker product, we can always express MCS in the vector-based fashion, but not vice versa.

Ii-B Compressive Learning

The objective of Compressive Learning is to create learning models that generate predictions from the compressed measurements or . This idea was first proposed in the early work of [15]

, where the authors train a classifier directly on the compressed measurements without the signal reconstruction step. It has been shown in this work that when the number of measurements approximates the dimensionality of the data manifold, accurate classifiers can be trained directly in the compressive domain. To achieve good classification performance, an extension of

[15] later showed that the number of measurements only depends on the intrinsic dimensionality of the data manifold [5].

A similar theoretical result was derived later in [9]

, which proves that the performance of a linear Support Vector Machine classifier trained on compressed measurements is equivalent to the best linear threshold classifier operating in the original signal domain, given the Distance-Preserving Property holds for the sensing matrices. Asymptotic behavior and sensing operator optimization in CL models of signals described by Gaussian Mixture Model have also been proposed in

[35, 36].

The idea of using past measurements as a prior to jointly learn better sensing matrices and the classifier was proposed in [4]. Since the advancement in stochastic optimization, linear classifiers and shallow learning models have been replaced with deep neural networks [29, 30], and separate optimization of the sensing operators and the classifiers have been replaced by the end-to-end training paradigm, which has been shown to significantly outperform predefined sensing operators when the compression rates are high [1, 22, 16, 49, 41].

Ii-C Multilinear Compressive Learning

While previous works in learning from compressed measurements adopt the signal acquisition model in Eq. (1), recently, the authors in [41] proposed Multilinear Compressive Learning (MCL), a framework that utilizes Eq. (2) as the signal acquisition model in order to retain the multidimensional structure of the original signal.

MCL consists of three components: the CS component described by Eq. (2), the Feature Synthesis (FS) component that transforms the measurements to tensor feature via:

(4)

and a task-specific neural network that operates on to generate prediction, i.e., the prediction is .

Since the transformation in Eq. (2) preserves the tensor structure of , MCL assumes the existence of a tensor subspace which captures the essential information in through , and from which discriminative features can be synthesized.

Similar to the end-to-end vector-based framework [1, 49], MCL incorporates a kind of weak prior that initializes and with energy-preserving values, i.e., the singular vectors corresponding to the largest singular values in mode- obtained from HOSVD of the training data. The task-specific network is initialized with the weights trained on uncompressed signals. The whole system is then optimized via stochastic gradient descent. As pointed out in the empirical analysis in [41], although simple, this initialization scheme contributes significantly to the performances of the system compared to the de facto random initialization scheme often utilized in deep neural networks [20].

Ii-D Knowledge Distillation

The term Knowledge Distillation (KD) was coined in [21], in which the authors proposed a neural network training technique that utilizes the prediction from a pre-trained high capacity network (the teacher network) to provide supervisory signals to a smaller network (the student network) along with labeled data. The intuition behind this training technique is that with higher capacity, it is easier for the teacher network to discover better data representation, and the form of knowledge provided via the teacher network’s prediction helps guide the student network to better solutions.

There have been several works investigating variants of this technique. For example, KD that involves multiple student and teacher networks [48] has been shown to be more robust than a single teacher-student pair. In other works [38, 46, 47]

, in addition to the predicted probabilities, knowledge coming from intermediate layers of the teacher network has also been proven to be useful for the student. While KD has often been considered in the context of model compression, i.e., to train low capacity models with better performances, this paradigm has also been successfully applied to distributed training with student and teacher networks having the same architecture

[2].

In our work, we propose to use KD as a method to progressively incorporate prior information into CL models. While in a general setting it is unclear how to select the immediate source and target layers to transfer knowledge from the teacher to the student, we will show next that in the context of CL, and especially in the MCL framework, pairing the intermediate teachers and students is easy since the learning system is modularized into different components with different functionalities.

Iii Proposed Methods

Iii-a Finding Prior Knowledge

Fig. 1: Illustration of the proposed algorithm: we first train prior-generating model . The knowledge in each component of is then progressively transferred to the MCL model. Lighter color blocks indicate components of while darker color ones indicate the corresponding components in MCL model.

A common constraint which exists in all CL models is that the sensing operator is linear or multilinear. This limits the capacity of CL models and the amount of information that can be captured from the original signal. Thus, in CL in general and MCL in particular, given a fixed structure of the Feature Synthesis (FS) component and the task-specific neural network , there is always an upper-bound on the capacity of the whole learning system, which makes the problem of learning from compressed measurements generally harder than other unconstrained learning tasks.

As mentioned in the Introduction Section, finding and incorporating prior knowledge is a recurring theme in CS community since good priors have been shown both theoretically and empirically to improve signal recovery and compression efficiency [33]. In case of CL, prior knowledge is less obvious. However, we may still formulate the following question: ´´What kind of prior knowledge do we know about the measurements and the types of features that are representative for the learning task?

Although we might not have direct knowledge of optimal or , we know as a prior knowledge that, given sufficient training data, a higher capacity neural network is expected to perform better than a less complex one when both networks are structurally similar. By structure here we mean that both networks have similar connectivity patterns and by higher capacity, we mean higher number of parameters and/or higher number of layers. The structural similarity ensures a feasible capacity comparison between networks since one cannot make such a guess about the learning ability, e.g., between a feed-forward architecture having a higher parameter count and a residual architecture having a lower parameter count. Therefore, we almost know for certain that given an MCL model with upper-bounded learning capacity, one can construct and train other learning systems with similar FS component and task-specific network , but having nonlinear sensing operators to achieve better learning performance, thus better and . The representations produced by higher capacity, nonlinear compressive learning models become our direct prior knowledge of the compressive domain and the feature space.

The advantage of the above presented approach to define prior knowledge in MCL is two-fold:

  • Since the priors are generated from another learning system trained on the same learning task, we incorporate completely data-dependent, task-specific priors into our model instead of hand-crafted priors.

  • By sharing the same structure of the FS component and task-specific neural network between the MCL model and the prior-generating model, the weights obtained from the latter provide a good initialization when optimizing the former.

The proposed approach reduces the problem of finding priors to the task of designing and training a nonlinear compressive learning model; hereupon we refer to it as the prior-generating model , which has no structural constraint. To construct and optimize , we propose to mirror the strategy in MCL [41]:

  • The nonlinear sensing component of , denoted as with parameters , has several layers to reduce the dimensionality of the original signal gradually. transforms the input signal to a compressed representation , i.e., , which has the same size as .

  • Similarly, the nonlinear FS component of , denoted as with parameters , has several layers to increase the dimensionality of gradually, producing features , which has the same dimensionality as the original signal. and

    together resemble the encoder and decoder of a nonlinear autoencoder.

  • Denote and () the -th training sample and its label. In order to initialize the weights and , we update and to minimize via stochastic gradient descent. The weights of task-specific network are initialized with weights minimizing with is the inference loss.

  • After the initialization step, , , and are updated altogether via stochastic gradient descent to minimize

1:Inputs:
2:     Labeled data set .
3:     Structure of prior-generating model , ,
4:     Structure of MCL model , ,
5:     Inference loss , Distillation loss
6:Training prior-generating model :
7:     Initialize with values from
8:         
9:     Initialize and with values from
10:         
11:     Obtain , , by solving
12:         
13:Training MCL model:
14:     After obtaining , , , they are fixed.
15:     Transfer sensing knowledge:
16:         Update by solving Eq. (5)
17:     Transfer feature synthesis knowledge:
18:         Update , by solving Eq. (6)
19:     Transfer inference knowledge & discriminative training:
20:         Update , , by solving Eq. (7)
21:Outputs: Parameters of MCL model: , ,
Algorithm 1

Training MCL Model via Knowledge Transfer Algorithm in Supervised Learning Setting

Iii-B Incorporating Prior Knowledge

Let us denote the CS, FS and task-specific neural network component of the MCL model as , and , having learnable parameters , and , respectively. It should be noted that and have the same functional form as and while possesses a multilinear form as in Eq. (2).

After training the prior-generating model , we perform progressive KD to transfer the knowledge of to the MCL model. During this phase, the parameters of , i.e., , , and are fixed. Since the teacher () and the student (MCL) model share the same modular structure which has three distinct components, training MCL with prior knowledge given by consists of three stages:

  • Stage 1 (Transferring the sensing knowledge from to ): during this stage, the weights of the sensing component in MCL are obtained by optimizing the following criterion:

    (5)

    The purpose of this stage is to enforce the sensing component in MCL to mimic that of the prior-generating model .

  • Stage 2 (Transferring the feature synthesis knowledge from to ): during this stage, we aim to optimize and in MCL to synthesize similar features produced by the prior-generating model. Before the optimization process, the weights of the sensing component () are initialized with values obtained from the 1st stage, and the weights of the feature synthesis component are initialized with values from since in MCL and in have the same functional form. After the initialization step, both sensing () and feature synthesis () components in MCL are updated together to minimize the following criterion:

    (6)
  • Stage 3 (Transferring the inference knowledge from to and discriminative training ): in this final stage, the MCL model is trained to minimize the inference loss as well as the difference between its prediction and the prediction produced by the prior-generating model . That is, the minimization objective in this stage is:

    (7)

    where and denote the inference loss and distillation loss, respectively. is a hyper-parameter that allows the adjustment of distillation loss. The specific form of and are chosen depending on the inference problem. For example, in classification problems, we can select cross-entropy function for

    and Kullback-Leibler divergence for

    while in regression problems, and can be mean-squared-error or mean-absolute-error function. To avoid lengthy representation, Eq. (7) omits the parameters of each component. Here we should note that before optimizing Eq. (7), parameters and of the sensing and synthesis component in MCL are initialized with values obtained from the previous stage, while parameters of the task-dependent neural network in MCL is initialized with values from in the prior-generating model .

The pseudo-code and the illustration of our proposed algorithm to train MCL model via progressive knowledge transfer is presented in Algorithm 1 and Figure 1, respectively.

To use KD and the prior-generating model , a simpler and more straightforward approach is to directly distill the predictions of to the MCL model as in the 3rd stage described above, skipping the 1st and 2nd stages. However, by gradually transferring knowledge from each component of the prior-generating model to the MCL model, we argue that the proposed training scheme directly incorporates the knowledge of a good compressive domain and feature space, facilitating the MCL model to mimic the internal representations of its teacher, which is better than simply imitating its teacher’s predictions. To validate our argument, we provide empirical analysis of the effects of each training stage in Section IV-F.

The aforementioned progressive knowledge transfer scheme also allows us to directly use the predictions produced by the teacher model in the 3rd stage, instead of the “softened” predictions as in the original work [21], eliminating the need to select the associated temperature hyper-parameter to soften the teacher’s predictions.

Iii-C Semi-supervised Learning Extension

It should be noted that without sufficient labeled data, training a high-capacity prior-generating model as proposed in Section III-A

can lead to an over-fitted teacher, which might provide misleading supervisory information to the MCL model. In addition, a limited amount of data might also prevent effective knowledge transfer from the teacher to the student model. In certain scenarios, labeled data is scarce, however, we can easily obtain a large amount of data coming from the same distribution without labels. Semi-supervised learning refers to the learning paradigm that takes advantages of unlabeled data, usually available in abundance, along with a limited amount of labeled data. Here we also describe a semi-supervised adaptation for the above MCL model training technique to remedy possible over-fitting cases and improve generalization of both prior-generating and MCL models in classification tasks. To this end, unlabeled data is utilized to both initialize, and then optimize the weights of the prior-generating model

via an incremental self-labeling procedure. Subsequently, when transferring knowledge to the MCL model, the class predictions of on unlabeled data are used as hard labels in the inference loss term, while its probability predictions on unlabeled data are used as soft labels in the distillation loss term.

Let us denote the labeled training set and the unlabeled training set. To take advatange of , we propose the following modifications to the training procedure of prior-generating model :

  • Initialization of and : the weights of the sensing () and feature synthesis () component are initialized with values obtained from minimizing the reconstruction error on , i.e., .

  • Incremental optimization of via self-labeling: after the initialization step, all parameters of the prior-generating model are optimized with respect to the inference loss, which is calculated on the enlarged labeled set :

    (8)

    Initially, the enlarged labeled set is formed from the labeled data, i.e., . After every backpropagation epochs, is augmented with those data instances (with their predicted labels) in that have the most confident predictions from the current , given a probability threshold , i.e., :

    (9)

    After the enlargement of with the most confident instances, they are removed from the unlabeled set , i.e., . The training terminates when the enlargement of stops, i.e., .

Self-labeling is a popular technique in semi-supervised algorithms. While there are many sophisticated variants of this technique [42], the simple modifications proposed above work well as illustrated in our empirical study in Section IV-G.

Given a prior-generating model trained on , in order to adapt the progressive knowledge transfer algorithm proposed in Section III-B to the semi-supervised setting, we propose to replace the objectives in Eq. (5), (6) and (7) with the following objectives respectively:

(10)

and

(11)

and

(12)

where in (12), denotes the class label predicted by the prior-generating model for .

A summary of our proposed algorithm in the semi-supervised setting is presented in Algorithm 2.

1:Inputs:
2:     Labeled data set .
3:     Unlabeled data set .
4:     Structure of prior-generating model , ,
5:     Structure of MCL model , ,
6:     Inference loss , Distillation loss
7:     Number of backpropagation iteration
8:     Confidence probability threshold
9:Training prior-generating model :
10:     Initialize with values from
11:         
12:     Initialize and with values from
13:         
14:     Let
15:     while True do
16:         Update , , by solving Eq. (8) for iterations
17:         Find set of confident instances according to Eq. (9)
18:         if then
19:            break
20:         Augment enlarged labeled set
21:         Reduce unlabeled set
22:Training MCL model:
23:     After obtaining , , , they are fixed.
24:     Transfer sensing knowledge:
25:         Update by solving Eq. (10)
26:     Transfer feature synthesis knowledge:
27:         Update , by solving Eq. (11)
28:     Transfer inference knowledge & discriminative training:
29:         Update , , by solving Eq. (12)
30:Outputs: Parameters of MCL model: , ,
Algorithm 2 Training MCL Model via Knowledge Transfer Algorithm in Semi-supervised Learning Setting

Iv Experiments

In this Section, we provide a detailed description and results of our empirical analysis which demonstrates the advantages of the proposed algorithms that incorporate data-dependent prior knowledge into the training procedure of MCL models, compared to [41]. For this purpose, we provide various comparisons between the proposed algorithm and the original algorithm proposed in [41] in the supervised learning setting. The experiments are designed to benchmark the learning performances and the quality of signal representation in compressive domain produced by the two competing algorithms, with increasing difficulty in the learning tasks. In addition, we also study the significance of prior-generating model and different knowledge transfer stages via different ablation experiments. Lastly, we demonstrate the necessity of the proposed modifications presented in Section III-C to our algorithm in the semi-supervised learning setting, i.e., when the databases are large, but only limited amounts of labeled data exist.

Iv-a Datasets

We conducted experiments on the object classification and face recognition tasks of the following datasets:

  • CIFAR dataset [27] is a color (RGB) image dataset used for evaluating object recognition methods. The dataset consists of images for training and images for testing with resolution pixels. In our work, CIFAR-10 refers to the -class objection recognition task in which each individual image has a single class label coming from different categories. A more fine-grained and difficult classification task also exists in CIFAR dataset with each image having a label coming from different categories, which we denote as CIFAR-100. Since there is no validation set in the original database, in our experiments, we randomly selected images from the training set of CIFAR-10 and CIFAR-100 for validation and only trained the algorithms on images.

  • CelebA: CelebA [28] is a large-scale human face image dataset with more than images at different resolutions from more than identities. In our experiment, we created three versions of CelebA with increasing difficulties by increasing the set of identities to be recognized: CelebA-100, CelebA-200, and CelebA-500 having , , and identities respectively. Here we note that CelebA-100 is a subset of CelebA-200, and CelebA-200 is a subset of CelebA-500.

  • To study the performances of the proposed algorithm in semi-supervised settings, we also created CIFAR-10S, CIFAR-100S, CelebA-500S, which have the same number of training instances as CIFAR-10, CIFAR-100, and CelebA-500, respectively, but only 20% of them are labeled. The test sets in CIFAR-10S, CIFAR-100S and CelebA-500S are the same as those in CIFAR-10, CIFAR-100, and CelebA-500.

The information of all datasets used in our experiments is summarized in Table I.

Dataset Input Dimension Output Dimension #Labeled Train #Unlabeled Train #Validation #Test
CIFAR-10
CIFAR-100
CelebA-100
CelebA-200
CelebA-500
CIFAR-10S
CIFAR-100S
CelebA-500S

TABLE I: Datasets Information

Iv-B Experiment Protocols

In this Section, to differentiate between MCL models trained by different algorithms, we use the abbreviation MCL to refer to the original algorithm proposed in [41], MCLwP to refer to our proposed algorithm that trains MCL models with a prior-generating model with labeled data only, and MCLwP-S to refer to our proposed algorithm that can take advantages of additional unlabeled data.

Regarding the architectural choices in MCL, MCLwP, MCLwP-S, we adopted the AllCNN architecture [41] for the task-specific neural network component in all algorithms. In MCL, the CS and FS components both perform multilinear transformation while in MCLwP and MCLwP-S, the CS component performs multilinear transformation and the FS component performs nonlinear transformation which consists of both convolution and multilinear projection. The exact network architecture used for each algorithm can be found in our publicly available implementation222https://github.com/viebboy/MultilinearCompressiveLearningWithPrior.

To compare the learning performances at different measurement rates, we have conducted experiments at four measurement rates , , and , corresponding to the following configurations of : , , , and , respectively.

Since all learning tasks are classification tasks, we represented the labels with one-hot encoding and used cross-entropy function and symmetric Kullback-Leibler divergence for the inference loss

and distillation loss , respectively. For each experiment configuration, we performed three runs, and the median values of accuracy on the test set are reported. For , the hyper-parameter that controls the amount of distillation loss in Eq. (7), we set it equal to . Confidence probability threshold in semi-supervised learning experiments was selected from the set .

Regarding stochastic optimization protocol, we used ADAM optimizer [26] with the following learning rate schedule , changing at epochs and . Each stochastic optimization procedure was conducted for epochs in total. Max-norm constraint with a value of was used to regularize the parameters in all networks. No data preprocessing was conducted, except the scaling of all pixel values to . During stochastic optimization, we performed random flipping on the horizontal axis and image shifting within of the spatial dimensions to augment the training set. In all experiments, the final model weights, which are used to measure the performance on the test sets, are obtained from the epoch, which has the highest validation accuracy.

Iv-C Comparisons with MCL [41]

Measurements Models CIFAR-10 CIFAR-100 CelebA-100 CelebA-200 CelebA-500
MCL [41]
MCLwP (our)
(MCLwPMCL)
MCL [41]
MCLwP (our)
(MCLwPMCL)
MCL [41]
MCLwP (our)
(MCLwPMCL)
MCL [41]
MCLwP (our)
(MCLwPMCL)
Prior ()
Prior ()
Prior ()
Prior ()
TABLE II: Learning performances of MCL [41] and MCLwP (our proposed algorithm) measured on test set (accuracy in %). The last four rows show the performances of prior-generating model for reference

Table II shows the performances of MCL proposed by [41] and our proposed algorithm MCLwP. The last four rows of Table II present the learning performances of the corresponding prior-generating models. It is clear that with the presence of prior knowledge, our proposed algorithm outperforms MCL for most of the configurations in all five datasets.

For a simpler problem such as recognizing objects in CIFAR-10 dataset, the performance gaps between MCLwP and MCL are relatively small. However, when the complexity of the learning problem increases, i.e., when the number of objects or facial identities increases, the differences in performance between MCLwP and MCL are significant. This can be observed by looking at the last row of each measurement configuration, with a direction from left to right (the direction of increasing difficulties of the learning tasks). For example, at configuration , moving from CIFAR-10 to CIFAR-100, the improvement changes from to , while from CelebA-100 to Celeb-200, then to CelebA-500, the improvement changes from to and to , respectively.

The only setting that MCLwP performs slightly worse than MCL is in CIFAR-10 dataset at the lowest measurement rate (). By inspecting the prior-generating models’ performances in CIFAR-10 dataset, we can see that the prior-generating model at measurement has very high performance, and even performs far better than its counterpart at a higher measurement rate (). Thus, the reason we see the degradation in learning performance might be because there is a huge gap between the teacher’s and the student’s learning capacity that makes the student model unable to learn effectively. This phenomenon in KD has been observed previously in [31].

One might make an assumption that lower numbers of measurements always associate with lower learning performances. This, however, is not necessarily true for our prior-generating models having nonlinear CS component with different numbers of downsampling layers. In fact, in our prior-generating models, we use two max-pooling layers to reduce the spatial dimensions for configuration while only one for configuration . Although better performances for each measurement configuration can be achieved by carefully adjusting the corresponding teacher’s capacity as in [31], it is sufficient for us to use a simple design pattern of autoencoder in order to demonstrate the effectiveness of the proposed MCLwP.

Iv-D Effects of Learning Capacity in Prior-Generating Models

Measurements Models CIFAR-10 CIFAR-100 CelebA-100 CelebA-200 CelebA-500
MCLwP
MCLwP⁢*
MCLwP
MCLwP⁢*
MCLwP
MCLwP⁢*
MCLwP
MCLwP⁢*
Prior
Prior
Prior
Prior
Prior
Prior
Prior
Prior
TABLE III: Learning performances of our proposed algorithm trained with two different prior-generating models ( and ). denotes the higher-capacity one. The last eight rows show the performances of the prior-generating models for reference.

While individual tweaking of each prior-generating model’s topology requires elaborate experimentation and is out of the scope of this work, we still conducted a simple set of experiments to study the overall effects when changing the prior-generating models’ capacity. In particular, we increased the capacity of the teacher models () in Section IV-C by adding more convolution layers in the CS and FS components. The set of higher-capacity teachers are denoted as , and the resulting student models are denoted as MCLwP⁢*.

Table III shows the learning performances of MCLwP in comparison with MCLwP⁢*, and in comparison with . While there are clear improvements in the learning performance of the teachers when we switch from to , we observe mixed behaviors in the corresponding student models. Here we should note that this phenomenon is expected since different measurement configurations in different datasets would require different adjustments (increase or decrease) of the teacher’s capacity to ensure the most efficient knowledge distillation. As observed in [31], the distribution of a student model’s performance with respect to different teacher models’ capacity has a bell shape. Thus, Table III can act as a guideline whether to increase or decrease the capacity of the prior-generating model in each configuration to maximize the learning performance of its student.

For example, in CIFAR-10 and CIFAR-100 dataset at measurement , increasing the teacher’s capacity from to leads to further degradation in the student’s performances; thus we should lower the capacity of to move toward the bell curve’s peak. On the other hand, in CIFAR-100 at or in CelebA-200 at , we should further upgrade the capacity of to possibly obtain better performing student models. As mentioned previously, this type of empirical hill climbing for each measurement configuration requires extensive experiments, however, might be necessary for certain practical applications.

Iv-E Quality of Compressive Domain

Measurements Models CIFAR-10 CIFAR-100 CelebA-100 CelebA-200 CelebA-500
#neighbors k=5
MCL [41]
MCLwP (our)
MCL [41]
MCLwP (our)
#neighbors k=20
MCL [41]
MCLwP (our)
MCL [41]
MCLwP (our)
TABLE IV: K-nearest neighbor performances (test accuracy in %) calculated using compressed measurements produced by MCL [41] and MCLwP (our proposed algorithm)

Although both MCL and MCLwP optimize the compressive learning models in an end-to-end manner, and there is no explicit loss term that regulates the compressive domain, it is still intuitive to expect models with better learning performances to possess better compressed representation at the same measurement rate. In order to quantify the representation produced by the competing algorithms in the compressive domain, we performed K-Nearest-Neighbor classification using the compressed representation at and after training MCL and MCLwP. Table IV shows the learning performances with two different neighbor values ( and ).

It is clear that MCLwP outperforms MCL in the majority of configurations. The performance gaps between MCLwP and MCL are more significant in facial image recognition tasks than in object recognition tasks. Here we should note that Euclidean distance was used to measure the similarity between data points, which might not be the optimal metric which can entirely reflect the semantic similarity and the quality of the compressive domain, especially when our compressive domains possess tensor forms that also encode spatial information. Besides, we cannot compare the performances of K-Nearest-Neighbor on a dataset across different measurements since measurements having larger spatial dimensions potentially retain more spatial variances, and Euclidean distance becomes less effective when measuring the similarity. For example, we can observe a decrease in performance on the CIFAR-10 and CIFAR-100 datasets, although the number of measurements increases from

to . For this reason, we did not study the performances of K-Nearest-Neighbor at higher measurements.

Iv-F Effects of Prior Knowledge

Measurements Models CIFAR-10 CIFAR-100 CelebA-100 CelebA-200 CelebA-500
MCLw/oP
MCLwP
MCLw/oP
MCLwP
MCLw/oP
MCLwP
MCLw/oP
MCLwP
TABLE V: Learning performances with (MCLwP) and without (MCLw/oP) the presence of prior knowledge

In order to study the effects of exploiting prior knowledge, we conducted two sets of experiments. While MCL and MCLwP models share the same structures of the CS and task-specific neural network component, there are architectural differences in the FS components: in MCL, the FS component performs multilinear transformation while in MCLwP, the synthesis step consists of both convolution and multilinear operations. Thus, in the first set of experiments, we aim to remove the architectural differences between the MCL and MCLwP models in the FS component that can potentially affect the quantification of prior knowledge. This is done by training the architectures which are trained by MCLwP in a similar manner as proposed in [41], i.e., without prior knowledge: we first initialized the CS and FS components by minimizing the reconstructed error via stochastic optimization, then trained the whole model with respect to the inference loss. This set of models are denoted as MCLw/oP and the comparisons between MCLwP and MCLw/oP are presented in Table V. Although we observe mixed behaviors at measurement , it is obvious that the presence of prior knowledge leads to improved learning performances without any architectural differences.

In the second set of experiments, we studied the effect of different knowledge transfer stages in MCLwP by either skipping or performing a particular knowledge transfer stage. This setup leads to different variants of the training procedure, which are presented in Table VI. Here we should note that when skipping the last transfer stage, we only discarded the distillation loss term and still trained the models with respect to the inference loss, instead of discarding the entire 3rd stage.

The first row of each dataset shows the results when all knowledge transfer activities are skipped. In other words, the architectures are initialized with a standard neural network initialization technique [19] and trained only with the inference loss. It is clear that this training procedure produces the worst performing models.

The next three rows in each dataset represent the cases where we performed only one of the knowledge transfer stages. In this scenario, conducting only the 2nd stage has noticeably better results than the other two cases, indicating the importance of the 2nd knowledge transfer stage.

Regarding the cases when one of the stages is skipped, we can also observe a homogeneous phenomenon across different measurements and different datasets that skipping the 1st stage leads to the least degradation. In fact, with this training procedure, we can obtain performances relatively close to the original MCLwP. During the 2nd knowledge transfer stage, the FS component () is updated in conjunction with the CS component () to mimic the features synthesized by the prior-generating model, which might imply an implicit knowledge transfer from to , thus might explain why skipping the 1st transfer stage only leads to minor degradations in performance. Overall, performing all three knowledge transfer stages yields the best results.

Transfer
Transfer
Transfer
CIFAR-10
CIFAR-100
CelebA-100
CelebA-200
CelebA-500
TABLE VI: Learning performances (test accuracy in %) of our proposed algorithm (MCLwP) when different knowledge transfer stages are skipped. The checkmark indicates when a particular transfer stage was conducted

Iv-G Semi-supervised Learning

Measurements Models CIFAR-10S CIFAR-100S CelebA-500S
MCL
MCLwP
MCLwP-S
MCL
MCLwP
MCLwP-S
MCL
MCLwP
MCLwP-S
MCL
MCLwP
MCLwP-S
Prior
Prior
Prior
Prior
Prior
Prior
Prior
Prior
TABLE VII: Learning performances in Semi-supervised Setting of MCL [41], MCLwP (trained with labeled data only), and MCLwP-S (trained with labeled and unlabeled data). The last eight rows show the performances of the corresponding prior-generating models ( and which provide prior knowledge for MCLwP and MCLwP-S, respectively.)

Finally, we present the empirical results in the semi-supervised setting. Here we should note that although the total amount of data is large, only a small fraction has labels for training. The competing algorithms include MCL [41] and MCLwP, both of which were trained with the labeled data only, and MCLwP-S, the semi-supervised extension of MCLwP which takes advantages of the unlabeled data in addition to the labeled set. We denote and the corresponding prior-generating models of MCLwP and MCLwP-S, respectively. The results are shown in Table VII.

It is clear that without sufficient data, the prior-generating models () cannot effectively transfer the knowledge to their student models (MCLwP) as can be seen by the inferior performances of MCLwP compared to MCL in half of the cases. However, when unlabeled data is utilized during the training process as proposed in our MCLwP-S algorithm, it is obvious that not only the prior-generating models () improve in performance but also their student models (MCLwP-S). The enhancement of the knowledge transfer process (via additional data) and prior-generating models (via self-labeling procedure) results in MCLwP-S having the best performances among the competing algorithms.

V Conclusions

In this work, we proposed a novel methodology to find and incorporate data-dependent prior knowledge into the training process of MCL models. In addition to the traditional supervised learning setting, we also proposed a semi-supervised adaptation that enables our methodology to take advantage of unlabeled data that comes from similar distributions of the signal of interest. Although we limited our investigation to MCL framework, the proposals presented in this work are sufficiently generic to be applicable to any CL systems. With extensive sets of experiments, we demonstrated the effectiveness of our algorithms to train MCL models in comparison with the previously proposed algorithm and provided insights into different aspects of the proposed methodologies.

Vi Acknowledgement

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR). This publication reflects the authors’ views only. The European Commission is not responsible for any use that may be made of the information it contains.

References

  • [1] A. Adler, M. Elad, and M. Zibulevsky (2016) Compressed learning: a deep neural network approach. arXiv preprint arXiv:1610.09615. Cited by: §I, §I, §II-B, §II-C.
  • [2] R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton (2018) Large scale distributed neural network training through online distillation. arXiv preprint arXiv:1804.03235. Cited by: §II-D.
  • [3] Y. August, C. Vachman, Y. Rivenson, and A. Stern (2013) Compressive hyperspectral imaging by random separable projections in both the spatial and the spectral domains. Applied optics 52 (10), pp. D46–D54. Cited by: §I.
  • [4] P. K. Baheti and M. A. Neifeld (2008) Adaptive feature-specific imaging: a face recognition example. Applied optics 47 (10), pp. B21–B31. Cited by: §I, §II-B.
  • [5] R. G. Baraniuk and M. B. Wakin (2009) Random projections of smooth manifolds. Foundations of computational mathematics 9 (1), pp. 51–77. Cited by: §II-B.
  • [6] D. Baron, M. B. Wakin, M. F. Duarte, S. Sarvotham, and R. G. Baraniuk (2005) Distributed compressed sensing. Cited by: §I.
  • [7] Y. Bengio, A. Courville, and P. Vincent (2013) Representation learning: a review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35 (8), pp. 1798–1828. Cited by: §I.
  • [8] C. F. Caiafa and A. Cichocki (2013) Multidimensional compressed sensing and their applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 3 (6), pp. 355–380. Cited by: §I.
  • [9] R. Calderbank and S. Jafarpour (2012) Finding needles in compressed haystacks. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3441–3444. Cited by: §I, §II-B.
  • [10] E. J. Candes, J. K. Romberg, and T. Tao (2006) Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences 59 (8), pp. 1207–1223. Cited by: §I.
  • [11] E. J. Candès and M. B. Wakin (2008) An introduction to compressive sampling [a sensing/sampling paradigm that goes against the common knowledge in data acquisition]. IEEE signal processing magazine 25 (2), pp. 21–30. Cited by: §I.
  • [12] V. Cevher, A. Sankaranarayanan, M. F. Duarte, D. Reddy, R. G. Baraniuk, and R. Chellappa (2008) Compressive sensing for background subtraction. In

    European Conference on Computer Vision

    ,
    pp. 155–168. Cited by: §I.
  • [13] A. Charles, M. S. Asif, J. Romberg, and C. Rozell (2011) Sparsity penalties in dynamical system estimation. In 2011 45th annual conference on information sciences and systems, pp. 1–6. Cited by: §I.
  • [14] M. A. Davenport, P. Boufounos, M. B. Wakin, R. G. Baraniuk, et al. (2010) Signal processing with compressive measurements.. J. Sel. Topics Signal Processing 4 (2), pp. 445–460. Cited by: §I.
  • [15] M. A. Davenport, M. F. Duarte, M. B. Wakin, J. N. Laska, D. Takhar, K. F. Kelly, and R. G. Baraniuk (2007) The smashed filter for compressive classification and target recognition. In Computational Imaging V, Vol. 6498, pp. 64980H. Cited by: §I, §I, §II-B.
  • [16] A. Değerli, S. Aslan, M. Yamac, B. Sankur, and M. Gabbouj (2018) Compressively sensed image recognition. In 2018 7th European Workshop on Visual Information Processing (EUVIP), pp. 1–6. Cited by: §I, §II-B.
  • [17] D. L. Donoho et al. (2006) Compressed sensing. IEEE Transactions on information theory 52 (4), pp. 1289–1306. Cited by: §I.
  • [18] Y. C. Eldar and M. Mishali (2009) Robust recovery of signals from a structured union of subspaces. IEEE Transactions on Information Theory 55 (11), pp. 5302–5316. Cited by: §I.
  • [19] X. Glorot and Y. Bengio (2010) Understanding the difficulty of training deep feedforward neural networks. In

    Proceedings of the thirteenth international conference on artificial intelligence and statistics

    ,
    pp. 249–256. Cited by: §IV-F.
  • [20] K. He, X. Zhang, S. Ren, and J. Sun (2016) Identity mappings in deep residual networks. In European conference on computer vision, pp. 630–645. Cited by: §II-C.
  • [21] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §I, §II-D, §III-B.
  • [22] B. Hollis, S. Patterson, and J. Trinkle (2018) Compressed learning for tactile object recognition. IEEE Robotics and Automation Letters 3 (3), pp. 1616–1623. Cited by: §I, §II-B.
  • [23] P. Irarrazabal and D. G. Nishimura (1995) Fast three dimensional magnetic resonance imaging. Magnetic resonance in medicine 33 (5), pp. 656–662. Cited by: §I.
  • [24] L. Kang and C. Lu (2009) Distributed compressive video sensing. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1169–1172. Cited by: §I.
  • [25] M. A. Khajehnejad, W. Xu, A. S. Avestimehr, and B. Hassibi (2011-05) Analyzing weighted minimization for sparse recovery with nonuniform sparse models. IEEE Transactions on Signal Processing 59 (5), pp. 1985–2001. External Links: Document, ISSN 1053-587X Cited by: §I.
  • [26] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §IV-B.
  • [27] A. Krizhevsky and G. Hinton (2009) Learning multiple layers of features from tiny images. Technical report Citeseer. Cited by: 1st item.
  • [28] Z. Liu, P. Luo, X. Wang, and X. Tang (2015-12) Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), Cited by: 2nd item.
  • [29] S. Lohit, K. Kulkarni, P. Turaga, J. Wang, and A. C. Sankaranarayanan (2015) Reconstruction-free inference on compressive measurements. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops

    ,
    pp. 16–24. Cited by: §I, §II-B.
  • [30] S. Lohit, K. Kulkarni, and P. Turaga (2016)

    Direct inference on compressive measurements using convolutional neural networks

    .
    In 2016 IEEE International Conference on Image Processing (ICIP), pp. 1913–1917. Cited by: §I, §II-B.
  • [31] S. Mirzadeh, M. Farajtabar, A. Li, and H. Ghasemzadeh (2019) Improved knowledge distillation via teacher assistant: bridging the gap between student and teacher. arXiv preprint arXiv:1902.03393. Cited by: §IV-C, §IV-C, §IV-D.
  • [32] P. Mohassel and Y. Zhang (2017) Secureml: a system for scalable privacy-preserving machine learning. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 19–38. Cited by: §I.
  • [33] J. F. Mota, N. Deligiannis, and M. R. Rodrigues (2017) Compressed sensing with prior information: strategies, geometry, and bounds. IEEE Transactions on Information Theory 63 (7), pp. 4472–4496. Cited by: §I, §III-A.
  • [34] V. M. Patel, G. R. Easley, D. M. Healy Jr, and R. Chellappa (2010) Compressed synthetic aperture radar. IEEE Journal of selected topics in signal processing 4 (2), pp. 244–254. Cited by: §I.
  • [35] H. Reboredo, F. Renna, R. Calderbank, and M. R. Rodrigues (2013) Compressive classification. In 2013 IEEE International Symposium on Information Theory, pp. 674–678. Cited by: §I, §II-B.
  • [36] H. Reboredo, F. Renna, R. Calderbank, and M. R. Rodrigues (2013) Projections designs for compressive classification. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 1029–1032. Cited by: §I, §II-B.
  • [37] F. Renna, R. Calderbank, L. Carin, and M. R. Rodrigues (2014) Reconstruction of signals drawn from a gaussian mixture via noisy compressive measurements. IEEE Transactions on Signal Processing 62 (9), pp. 2265–2277. Cited by: §I.
  • [38] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio (2014) Fitnets: hints for thin deep nets. arXiv preprint arXiv:1412.6550. Cited by: §II-D.
  • [39] V. Stanković, L. Stanković, and S. Cheng (2009) Compressive image sampling with side information. In 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 3037–3040. Cited by: §I.
  • [40] M. Stojnic, F. Parvaresh, and B. Hassibi (2009) On the reconstruction of block-sparse signals with an optimal number of measurements. IEEE Transactions on Signal Processing 57 (8), pp. 3075–3085. Cited by: §I.
  • [41] D. T. Tran, M. Yamac, A. Degerli, M. Gabbouj, and A. Iosifidis (2019) Multilinear compressive learning. arXiv preprint arXiv:1905.07481. Cited by: §I, §I, §I, §II-B, §II-C, §II-C, §III-A, §IV-B, §IV-B, §IV-C, §IV-C, §IV-F, §IV-G, TABLE II, TABLE IV, TABLE VII, §IV.
  • [42] I. Triguero, S. García, and F. Herrera (2015) Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems 42 (2), pp. 245–284. Cited by: §III-C.
  • [43] M. Trocan, T. Maugey, J. E. Fowler, and B. Pesquet-Popescu (2010) Disparity-compensated compressed-sensing reconstruction for multiview images. In 2010 IEEE International Conference on Multimedia and Expo, pp. 1225–1229. Cited by: §I.
  • [44] N. Vaswani and W. Lu (2010) Modified-cs: modifying compressive sensing for problems with partially known support. IEEE Transactions on Signal Processing 58 (9), pp. 4595–4607. Cited by: §I.
  • [45] M. Yamac, M. Ahishali, N. Passalis, J. Raitoharju, B. Sankur, and M. Gabbouj (2019) Reversible privacy preservation using multi-level encryption and compressive sensing. arXiv preprint arXiv:1906.08713. Cited by: §I.
  • [46] J. Yim, D. Joo, J. Bae, and J. Kim (2017)

    A gift from knowledge distillation: fast optimization, network minimization and transfer learning

    .
    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133–4141. Cited by: §II-D.
  • [47] R. Yu, A. Li, V. I. Morariu, and L. S. Davis (2017) Visual relationship detection with internal and external linguistic knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1974–1982. Cited by: §II-D.
  • [48] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu (2018) Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320–4328. Cited by: §II-D.
  • [49] E. Zisselman, A. Adler, and M. Elad (2018) Compressed learning for image classification: a deep neural network approach. Processing, Analyzing and Learning of Images, Shapes, and Forms 19, pp. 1. Cited by: §I, §II-B, §II-C.