Sparse coding (SC) is a classical unsupervised feature extraction technique for finding concise representations of the data, which has been successfully applied to numerous areas across computer vision and pattern recognition. Conventional SC  aims to approximate vector-valued inputs by linear combinations of a few bases. Such bases correspond to patterns that physically represent elementary objects, and they compose a dictionary.
Conventional SC 
model suffers from the following two major problems: 1) the vectorization preprocess elementarily breaks apart the local proximity and destructs the object structures of images; and 2) the high computational complexity restricts its applications, thus small sizes of patches are usually used. Usually, the dictionary is overcomplete as the number of bases is larger than the dimension of the input image data. Therefore, the dictionary size is significantly large for high dimensional data, meaning that it requires prohibitive large number of computations for conventional SC.
However, existing approaches cannot solve the above two problems satisfactorily. Two kinds of SC models are proposed to preserve the spatial proximity of images, which are tensor sparse coding (TenSR) [2, 3] and convolutional sparse coding (CSC) [12, 13]. For TenSR models [2, 3], a series of separable dictionaries are adopted to approximate the structures in each mode of the input data. Though the sizes of the dictionaries are significantly reduced, the relationships among the modes are ignored. The object structures usually distribute across all modes of the data. For CSC models [12, 13], the dictionaries are used to capture local patterns, and the convolution operator is introduced to learn the shifting-invariant patterns. However, optimizing such models with the convolution operator are computational challenging. Moreover, each feature map (sparse representation) has nearly the same size as the input image, which is quite larger than conventional SC, and will increase the resources for storage and the computational complexity.
The basic idea that motivates us to address these challenges lies in two aspects: 1) tensor representation is able to preserve the local proximity and to capture the elementary object structures; and 2) we exploit the tensor-product operation under a novel algebraic framework where the tensor-linear combinations are used to approximate the images, instead of the tucker decomposition used in TenSR [2, 3] and convolution operation in CSC [12, 13]. For one aspect, tensor-product based on circular convolution operation, which can generate the data by shifted versions of bases. Fig. 1 shows the shifted versions generated from the tensor-product without storing them. For another aspect, the tensor-linear combination (see Definition 4) is a generalization of the standard linear combination. The number of required bases can be significantly reduced, which also reduces the computational complexity.
In this paper, we propose a novel sparse coding model, two-dimensional sparse coding (2DSC), in which the input images are represented as third-order tensors, and the tensor-linear combinations are used for data approximation. To solve the 2DSC problem, a novel alternating minimization algorithm is presented which includes a sparse coding learning step and a dictionary learning step. For sparse coding, we propose a new iterative shrinkage thresholding algorithm based on tensor-product, which is directly implemented in the tensor space. For dictionary learning, we show that it can be solved efficiently by transforming to a Lagrange dual problem in the frequency domain.
The rest of this paper is organized as follows: Section 3 introduces the notations and preliminary used in our paper. Section 4 presents the proposed 2DSC model, followed by the novelties of our model. In Section 5, an efficient alternating minimization algorithm for 2DSC is proposed. We demonstrate the effectiveness of our 2DSC model by simulation experiment and multi-spectral images denoising in Section 6. Finally, we conclude the paper in Section 7. To summarize, this paper makes the following contributions:
We propose a novel two-dimensional sparse coding model (2DSC) for image representation, which preserves the local proximity of pixels and elementary object structures of images. 2DSC is superior in dealing with high dimensional data.
We discuss the geometric properties of the dictionary returned by 2DSC, and show that there exists an equivalent sum space spanned by the corresponding vectorizations of the bases. Therefore, the dictionary of 2DSC has a stronger representation capability than that of the conventional SC.
We propose an efficient alternating minimization algorithm. For coefficient learning, a novel iterative shrinkage thresholding algorithm based on tensor-product is provided which exploits the optimization of mechanism in the tensor space. For dictionary learning, we convert it into a corresponding problem in the frequency domain and solve the dictionary by dual Lagrange which significantly reduces the number of variables to be optimized.
2 Related Work
In this section, we briefly review the related work on sparse coding models with consideration of spatial structures of the images, including sparse coding based on tensor representations [2, 3] and convolutional sparse coding [12, 13].
The first stream is the tensor-based sparse coding (TenSR) models [2, 3], which preserves the spatial structures of images by tensor representations. The data is represented based on tucker decomposition. Instead of using one dictionary as conventional SC, a series of separable dictionaries are adopted to model data, and each dictionary is corresponding to one dimension of the data. Therefore, the sizes of the dictionaries are significantly reduced due to the small size of each dimension compared with the sizes of the data. Though the sizes of dictionaries are significantly reduced, the relationships between the modes are ignored. The object structures are usually distributed across all dimensions of the data.
The second stream is the convolutional sparse coding (CSC) models [12, 13], which represents an image as the summation of convolutions of the feature maps and the corresponding filters. Convolution operator models the similarities of the proximal pixels, which further preserves the spatial structures of images. However, optimizing CSC models are computational challenging. Moreover, each feature map (sparse representation) has nearly the same size as the image, which is quite larger than conventional SC, and will increase the resources for storage and the computational complexity.
3 Notation and Preliminary
A third-order tensor is denoted as . The expansion of along the third dimension is represented as . The transpose of tensor is denoted as , where , and ,
, and the superscript “T” represents the transpose of matrices. The discrete Fourier transform (DFT) along the third dimension ofis denoted as .
For convenience, tensor spaces , , and are denoted as , , and , respectively. denotes the set . The and Frobenius norms of tensors are denoted as , and . Furthermore, we need the following definitions.
 The tensor-product between and is a tensor where , and denotes the circular convolution operation.
The tensor-product can be efficiently computed in the frequency domain as:
 The tensor-product has an equivalent matrix-product as:
where is the circular matrix of defined as follows:
 The tensor-linear combinations of the tensor bases with the corresponding tensor coefficient are defined as:
where with , and with .
The tensor-linear combination is a generalization of the linear combination.
The spanned tensor space by the tensor bases set of is defined as .
4 Problem Statement
4.1 Problem Formulation
Instead of preprocessing images into vectors, we represent images of size by , and propose a novel sparse coding model, named two-dimensional sparse coding (2DSC), as follows:
where is the tensor dictionary where each lateral slice is a basis, is the tensor coefficient. The parameter balances the approximation error and the sparsity of the tensor coefficients, and is the number of atoms. Conventional SC is a special case of (4.1) when .
4.2 Novelties of 2DSC
The proposed 2DSC is not a simply extension of conventional SC on the two-dimensional data, which has novel properties. The first one is the size of dictionary in 2DSC can be significantly reduced without damaging the reconstruction accuracy due to the tensor-linear combination. The second one is shifting invariant which means that the data can be generated from 2DSC model by the shifted versions of bases without explicitly storing them.
4.2.1 “Slim” Dictionary
 Tensor space can be generated by tensors from an orthogonal set.
Fig. 2 shows four kinds of sparse coding models for an image representation. In SC , space is generated by bases based on linear combination, while in 2DSC, can be spanned with only bases. In TenSR  and CSC , it is not easy to determine the number of bases for the image space. Though CSC  is also based on convolution operator, the sparse representations are nearly the size as the input images, which are quite larger than ours. Lemma 2 means an data can be generated by only elements with the same size based on tensor linear combination. However, elements are required based on linear combinations. With much fewer atoms for data representation can significantly reduce the computational complexity, which shows the potential applications of 2DSC model in high dimensional data.
4.2.2 Shifting Invariance
The tensor space spanned by defined in (3) is equivalent to a sum space of vector subspaces in .
where is the vectorizations of , the bases in are circular shifted versions of those in . and are the coefficients to the corresponding dictionaries and . Fig. 1 explicitly shows the shifted versions of a base generated by tensor-product if the image of helicopter is seen as a basis. The space generated by the tensor linear combination can be transferred to a sum space of vector spaces generated by the linear combination, which includes the shifted versions of the original atoms. Fig. 3 explicitly shows the equivalent linear combination and the shifted versions of a atom generated from tensor product by its twisted form of size and a coefficient.
5 Alternating Minimization Algorithm
Problem (4.1) is quite challenging due to the non-convex objective function and the convolution operator. Instead of transforming (4.1) into conventional SC formulation based on Lemma 1, we propose an efficient algorithm by alternately optimizing and in the tensor space, as shown in Algorithm 1.
5.1 Learning Tensor Coefficient
For clarity, we discuss how to solve the tensor sparse representation for an image of size , which is represented as .
Given the dictionary , solving the tensor sparse representations of images are converted to the following problem as:
where , , and . The size of the dictionary in (8) is significantly increased with the size of images, which also increase the computational complexity.
where stands for the data reconstruction term and stands for the sparsity constraint term . An iterative shrinkage algorithm is used to solve (9
), which can be rewritten as a linearized function around the previous estimationwith the proximal regularization and the non-smooth regularization. Thus, at the - th iteration, can be updated by
where is a Lipschitz constant, and is the gradient defined in the tensor space. Then, (10) is equivalent to
To solve (11), we firstly show w.r.t. the data reconstruction term :
Secondly, we discuss how to determine the Lipschitz constant in (11). For every , , we have
where the superscript “” represents conjugate transpose.
Thus the Lipschitz constant of used in our algorithm is .
Lastly, (11) can be solved by the proximal operator , where is the soft-thresholding operator .
5.2 Tensor Dictionary learning
For learning the dictionary while fixed , the optimization problem is:
where atoms are coupled together due to the circular convolution operator. Therefore, we firstly decompose (5.2) into nearly-independent problems (that are coupled only through the norm constraint) by DFT as follows:
Then, we adopt the Lagrange dual  for solving (5.2) in frequency domain. The advantage of Lagrange dual is that the number of optimization variables is , which is much smaller than in the primal problem (5.2).
To use the Lagrange dual algorithm, firstly, we consider the Lagrangian of (5.2):
where , is a dual variable, and .
5.3 Complexity Analysis
Table 1 shows the computational complexity of our proposed 2DSC and TenSR . The definitions of and can be found in . , and are the sizes of dictionaries in our model and TenSR . The computational complexity of TenSR  is higher than ours. As shown in , for 21168 patches of size , TenSR takes 189 seconds for tensor coefficient learning, while in our model, it only takes about 23 seconds. The reason is due to the faster computation of tensor-product in frequency domain, which divides the original large size problems into much smaller ones.
6.1 Dictionaries for Large Patches
We analyze the learned dictionaries from conventional SC and 2DSC with the same bases and other parameters settings. Patches of sizes () extracted from Natural Images data are normalized to have zero means. The sparsity regularizers is set to 0.1, and the number of bases are 64, which makes sure the dictionary for 2DSC is overcompleted but not for SC. Fig. 4 shows the learned dictionaries from 2DSC and conventional SC with different sizes of patches (). For small size of patches (), both models can learn the meaningful dictionaries (Gabor-like featues), such as edges, corners. For large size patches (), the performances of SC are significantly degenerated, even the bases are becoming sparsity. But 2DSC still can learn meaningful features.
6.2 Multi-spectral Image Denoising
We apply 2DSC on multispectral images – Columbia MSI Database . Each dataset contains 31 real-world images of size and is collected from to at
steps. We firstly to scale these images to [0, 255], and then add Gaussian white noise at different noise levels. In our 2DSC model, we extract patches of size from each noisy multi-spectral image, and save each patch into a tensor of size . Dictionary of size are randomly initialized and trained iteratively ( iterations). Then we use the learned dictionaries to denoise the MSI images. Parameters in our scheme are , for , respectively.
Table 2 shows the comparison results in terms of peak signal-to-noise ratio (PSNR) and structure similarity (SSIM)  . There are 5 state-of-the-art MSI denoising methods are involved, including band-wise KSVD (BwK-SVD) method , 3D-cube KSVD (3DK-SVD) method , LRTA , PARAFAC , and TenSR . As shown in Table 2, our 2DSC outperforms all the comparison algorithms for the evaluations by SSIM, which measures the structure consistency between the target image and the reference image. For PSNR, our 2DSC works much better on higher level noises, especially for and . For lower noise levels, LRTA  is the best one, ours and TenSR  are comparable.
We add CSC  in the visualization of the denoising performances. Due to the high memory required by CSC  for the high resolution images, we resize the spectral images of balloons into , and only considering the first five bands. Fig. 5 shows the denoising results with the noise level . It is easy to observe that our method achieves the best denoising results. Note that for CSC , we denoise each images separately which does not consider the information along the brands of spectral images. Filters of size are used, and the sparsity parameters are adjusted between [0.1, 10]. We guess the reason for the denoising results of CSC  may come from the small size of images. Due to the powerful representations of convolution operation, filters learn much noise information.
In this paper, we propose a novel tensor based sparse coding algorithm, which can learn an efficient tensor representations of images by much smaller size of dictionary compared with conventional SC. Moreover, an much more efficient algorithm for the tensor sparse coefficients learning in tensor space is proposed. The effectiveness of our model has been demonstrated by dictionary learning for large sizes of patches. A following up work  applies the 2DSC scheme to image clustering by incorporating a graph regularizer.
-  Wright, John and Ma, Yi and Mairal, Julien and Sapiro, Guillermo and Huang, Thomas S and Yan, Shuicheng, Sparse representation for computer vision and pattern recognition, Proceedings of the IEEE, vol. 98, no. 6, pp. 1031–1044, 2010.
-  Qi, Na and Shi, Yunhui and Sun, Xiaoyan and Yin, Baocai, TenSR: Multi-Dimensional Tensor Sparse Representation, IEEE CVPR, pp. 5916-2925, 2016.
-  Qi, Na and Shi, Yunhui and Sun, Xiaoyan and Wang, Jingdong and Yin, Baocai, Two dimensional synthesis sparse model, IEEE ICME, pp. 1-6, 2013.
-  Kernfeld, Eric and Aeron, Shuchin and Kilmer, Misha, Clustering multi-way data: a novel algebraic approach, arXiv preprint arXiv:1412.7056, 2014.
Kernfeld, Eric and Kilmer, Misha and Aeron, Shuchin, Tensor–tensor products with invertible linear transforms, Linear Algebra and its Applications, vol. 485, pp. 545-570, 2015.
Qiu, Qiang and Chellappa, Rama, Compositional dictionaries for domain adaptive face recognition, IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5152-5165, 2015.
-  Wong, Wai Keung and Lai, Zhihui and Xu, Yong and Wen, Jiajun and Ho, Chu Po, Joint tensor feature analysis for visual object recognition, IEEE transactions on cybernetics, vol. 45, no. 11, pp. 2425-2436, 2015.
-  Zhao, Qibin and Zhou, Guoxu and Zhang, Liqing and Cichocki, Andrzej, Tensor-variate gaussian processes regression and its application to video surveillance, IEEE ICASSP, pp. 1265-1269, 2014.
-  Lee, Honglak and Battle, Alexis and Raina, Rajat and Ng, Andrew Y, Efficient sparse coding algorithms, NIPS, pp. 801-808, 2006.
-  Ayyala, Deepak Nag, Least Angle Regression, LARS, vol. 2, no. 23, 2008.
-  Donoho, David L and Tsaig, Yaakov and Drori, Iddo and Starck, Jean-Luc. Sparse solution of underdetermined systems of linear equations by stagewise orthogonal matching pursuit, IEEE Transactions on Information Theory, vol. 58, no. 2, pp. 1094-1121, 2012.
-  Bristow, Hilton and Eriksson, Anders and Lucey, Simon, Fast convolutional sparse coding, IEEE CVPR, pp. 391-398, 2013.
-  Heide, Felix and Heidrich, Wolfgang and Wetzstein, Gordon, Fast and flexible convolutional sparse coding, IEEE CVPR, pp. 5135-5143, 2015.
-  Kilmer, Misha E and Braman, Karen and Hao, Ning and Hoover, Randy C, Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging, SIAM Journal on Matrix Analysis and Applications, vol. 34, no. 1, pp. 148-172, 2013.
-  Peng, Yi and Meng, Deyu and Xu, Zongben and Gao, Chenqiang and Yang, Yi and Zhang, Biao, Decomposable nonlocal tensor dictionary learning for multispectral image denoising, IEEE CVPR, pp. 2949–2956, 2014.
-  Maggioni, Matteo and Katkovnik, Vladimir and Egiazarian, Karen and Foi Alessandro, Nonlocal transform-domain filter for volumetric data denoising and reconstruction, IEEE transactions on image processing, vol. 22, no. 1, pp. 119-133, 2013.
-  Liu, Xuefeng and Bourennane, Salah and Fossati, Caroline, Denoising of hyperspectral images using the PARAFAC model and statistical performance analysis, IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 10, pp. 3717-3724, 2012.
-  Renard, Nadine and Bourennane, Salah and Blanc-Talon, Jacques, Denoising and dimensionality reduction using multilinear tools for hyperspectral images, IEEE Geoscience and Remote Sensing Letters, vol. 5, no. 2, pp. 138-142, 2008.
-  Elad, Michael and Aharon, Michal, Image denoising via sparse and redundant representations over learned dictionaries, IEEE Transactions on Image processing, vol. 15, no. 12, pp. 3736-3745, 2006.
-  Yasuma, Fumihito and Mitsunaga, Tomoo and Iso, Daisuke and Nayar, Shree K, Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum, IEEE Transactions on Image Processing, vol. 19, no. 9, pp. 2241-2253, 2010.
-  Wang, Zhou and Bovik, Alan C and Sheikh, Hamid R and Simoncelli, Eero P, Image quality assessment: from error visibility to structural similarity, IEEE transactions on image processing, vol. 13, no. 4, pp. 600-612, 2004.
-  Xu, Yangyang and Yin, Wotao, A fast patch-dictionary method for whole image recovery, arXiv:1408.3740, 2014.
-  Jiang, Fei and Liu, Xiao-Yang and Lu, Hongtao and Shen, Ruimin, Graph regularized tensor sparse coding for image representation, IEEE ICME, 2017.