Spectral Collaborative Filtering

08/30/2018 ∙ by Lei Zheng, et al. ∙ IFM Lab Peking University University of Illinois at Chicago 0

Despite the popularity of Collaborative Filtering (CF), CF-based methods are haunted by the cold-start problem, which has a significantly negative impact on users' experiences with Recommender Systems (RS). In this paper, to overcome the aforementioned drawback, we first formulate the relationships between users and items as a bipartite graph. Then, we propose a new spectral convolution operation directly performing in the spectral domain, where not only the proximity information of a graph but also the connectivity information hidden in the graph are revealed. With the proposed spectral convolution operation, we build a deep recommendation model called Spectral Collaborative Filtering (SpectralCF). Benefiting from the rich information of connectivity existing in the spectral domain, SpectralCF is capable of discovering deep connections between users and items and therefore, alleviates the cold-start problem for CF. To the best of our knowledge, SpectralCF is the first CF-based method directly learning from the spectral domains of user-item bipartite graphs. We apply our method on several standard datasets. It is shown that SpectralCF significantly outperforms state-of-the-art models. Code and data are available at <https://github.com/lzheng21/SpectralCF>.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

The effectiveness of recommender systems (RS) often relies on how well users’ interests or preferences can be understood and interactions between users and items can be modeled. Collaborative Filtering (CF) (Koren et al., 2009) is one of the widely used and prominent techniques for RS. The underlying assumption of the CF approach is that if a user shares a common item with another user , is also likely to be interested in other items liked by . Although CF has been successfully applied to many recommendation applications, the cold-start problem is considered as one of its major challenges (Koren et al., 2009). The problem arises when a user interacted with a very small number of items. Consequently, the user shares few items with other users, and effectively recommending for the user becomes a challenging task for RS.

Figure 1. A toy example of a user-item bipartite graph with edges representing observed user-item interactions. Red circles and green rectangles denote users and items, respectively.

If we formulate the relationships between users and items as a bipartite graph111

In this paper, we use the terminology ”graph” to refer to the graph/network structure of data and ”network” for the architecture of machine learning models.

, we argue that the connectivity information of the graph can play an important role for tackling the cold-start problem. For example, let us see a bipartite graph in Figure 1. A cold-start user only interacts with item . Since shares with user and user , as a result, three items (, and ) connected with or can all be recommended to by a CF-based model. However, a natural and important question arises: which one in the three items is the most reliable recommendation for ? The key to answer the question lies in the user-item connectivity information. In fact, if we take a look at the connections of the graph, it is clear that there is only one path existing between and (or ), while two paths connect to . Thus, compared with and , obviously, is a more reliable recommendation for .

However, existing CF-based methods, including model-based and memory-based approaches, often suffer from the difficulty of modeling the connectivity information. Previous model-based approaches, such as Matrix Factorization (MF) (Koren et al., 2009), are usually designed to approximate the direct connections (or proximities). However, indirect connectivity information hidden in the graph structures is rarely captured by traditional model-based approaches. For instance, it is formidable for them to model the number of paths between and in Figure 1. Whereas a number of memory-based approaches (Sarwar et al., 2001; Jamali and Ester, 2009) is introduced to model the connectivity information, these methods often rely on pre-defined similarity functions. However, in the real world, defining an appropriate similarity function suitable for diverse application cases is never an easy task.

Spectral graph theory (Shuman et al., 2013)

studies connections between combinatorial properties of a graph and the eigenvalues of matrices associated to the graph, such as the laplacian matrix (see Definition

2.4 in Section 2). In general, the spectrum of a graph focuses on the connectivity of the graph, instead of the geometrical proximity. To see how does the spectral domain come to help for recommendations and better understand the advantages of viewing a user-item bipartite graph in the spectral perspective, let us revisit the toy example shown in Figure 1. For the bipartite graph

, we visually plot its vertices in one specific frequency domain. Although vertices do not come with coordinates, a popular way to draw them in a space is to use eigenvectors of a laplacian matrix associated with the graph to supply coordinates

(Spielman, 2007). Figure 2 shows that, compared with and , becomes closer to in the space222In spectral graph theory, smaller (or larger) eigenvalues of the associated laplacian matrix corresponds to lower- (or higher-) frequency domains. In Figure 1, we plot each vertex at the point , where indicates the value of the eigenvector of the laplacian matrix .. Thus, when transformed into the frequency domain, is revealed to be a more suitable choice than or for . The underlying reason is that the connectivity information of the graph has been uncovered in the frequency domain, where the relationships between vertices depend on not only their proximity but also connectivity. Thus, exploiting the spectrum of a graph can help better explore and identify the items to be recommended.

Inspired by the recent progress (Kipf and Welling, 2016; Defferrard et al., 2016) in node/graph classification methods, we propose a spectral graph theory based method to leverage the broad information existing in the spectral domain to overcome the aforementioned drawbacks and challenges. Specifically, to conquer the difficulties (see Section 3.3) of directly learning from the spectral domain for recommendations, we first present a new spectral convolution operation (see Eq. (10)), which is approximated by a polynomial to dynamically amplify or attenuate each frequency domain. Then, we introduce a deep recommendation model, named Spectral Collaborative Filtering (SpectralCF), built by multiple proposed spectral convolution layers. SpectralCF directly performs collaborative filtering in the spectral domain.

Figure 2. Vertices of the bipartite graph in Figure 1 are plotted in a frequency domain. Note that the vertices not shown above are omitted for simplicity.

The key contributions of this work can be summarized as follows:

  • Novelty: To the best of our knowledge, it is the first CF-based method directly learning from the spectral domains of user-item bipartite graphs.

  • A deep recommendation model: We propose a new spectral convolution operation performing in the spectral domain. Stacked by multiple layers of the proposed spectral convolution operation, a deep recommendation model, named Spectral Collaborative Filtering (SpectralCF), is introduced.

  • Strong Performance: In the experiments, SpectralCF outperforms state-of-the-art comparative models. It is shown that SpectralCF effectively utilizes the rich information of connectivity existing in the spectral domain to ease the cold-start problem.

The rest of the paper is organized as follows. In Section 2, we provide preliminary concepts. Section 3 describes SpectralCF in detail. Experiments are presented in Section 4 to analyze SpectralCF and demonstrate its effectiveness compared with state-of-the-art techniques for RS. In Section 5, we give a short review of the works related to our study. Finally, conclusions are presented in Section 6.

2. Definitions and Preliminaries

In this section, we present the background and preliminaries of this study. Throughout the paper, we denote scalars by either lowercase or uppercase letters, vectors by boldfaced lowercase letters, and matrices by boldfaced uppercase letters. Unless otherwise specified, all vectors are considered to be column vectors. Let

denote an identity matrix, and

and denote matrices of ones and zeros, respectively. In addition, we define the following definitions in this paper as:

Definition 2.0.

(Bipartite Graph). A bipartite user-item graph with vertices and edges for recommendations is defined as , where and are two disjoint vertex sets of users and items. Every edge has the form where and and denotes that user has interacted with item in the training set.

Definition 2.0.

(Implicit Feedback Matrix). An implicit feedback matrix is a matrix defined as following:

(1)
Definition 2.0.

(Adjacent Matrix). For the bipartite graph , its corresponding adjacent matrix can be defined as:

(2)

where is an matrix.

Definition 2.0.

(Laplacian Matrix). The random walk laplacian matrix is defined as , where is the identity matrix and is the diagonal degree matrix defined as .

This paper focuses on the recommendation problem with implicit feedbacks, where we only observe whether a person has viewed/liked/clicked an item and do not observe explicit ratings. Let denote the set of all items liked by user and denote the remaining items. We define the recommendation problem which we study in this paper as the following:

Definition 2.0.

(Problem Definition). Given a user set and an item set , for each user who has liked/clicked/viewed an item set , we aim to recommend a ranked list of items from that are of interests to the user.

3. Proposed Model

In this section, we first describe the process of performing a

graph fourier transform

on a bipartite graph for recommendations. Then we propose to place a novel spectral convolution filter on vertices (users and items) of the bipartite graph to dynamically filter the contributions of each frequency component in the spectral domain. Later, a polynomial approximation is employed to overcome the shortcomings of the proposed convolution operation. Finally, with the approximate convolution operation, we introduce our final recommender system, named Spectral Collaborative Filtering, stacked by multiple spectral convolution layers.

3.1. Graph Fourier Transform

Definition 3.0.

(Graph Signal). Given any graph , where and are a vertex and an edge set, respectively, a graph signal is defined as a state vector over all vertices in the graph, where is the value of observed at the vertex of .

The classical fourier transform is defined as an expansion of a function in terms of the complex exponentials as:

(3)

where is an imaginary number, and the complex exponentials () form an orthonormal basis.

Analogously, the graph fourier transform is defined as an expansion of an observed graph signal in terms of the eigenvectors of the graph laplacian , and the eigenvectors serve as a basis in the spectral domain. Let us assume that a graph signal () is observed on a graph , we define the graph fourier transform and its inverse on as:

(4)

where , and denote the , and value of , and , respectively; denotes the eigenvector of ; represents a graph signal which has been transformed into the spectral domain. For simplicity, we rewrite Eq. (4) in the matrix form as and , respectively, where are eigenvectors of .

In particular, for a bipartite graph , assume that there are two types of graph signals: and , associated with user and item vertices, respectively. We transform them into the spectral domain and vice versa as :

(5)

3.2. Spectral Convolution Filtering

The broad information of graph structures exists in the spectral domain, and different types of connectivity information between users and items can be uncovered in different frequency domains. It is desirable to dynamically adjust the importance of each frequency domain for RS.

To this end, we propose a convolution filter, parameterized by , as into the spectral domain as:

(6)

where and are new graph signals on learned by the filter , and denotes eigenvalues of the graph laplacian matrix .

In Eq. (6), a convolution filter is placed on a spectral graph signal , and each value of is responsible for boosting or diminishing each corresponding frequency component. The eigenvector matrix in Eq. (6) is used to perform an inverse graph fourier transform.

3.3. Polynomial Approximation

Recall that we proposed a convolution operation, as shown in Eq. (6), to directly perform in the spectral domain. Although the filter is able to dynamically measure contributions of each frequency component for the purpose of recommendations, there are two limitations. First, as shown in Eq. (6), the learning complexity of the filter is , where

is the number of vertices. That is, unlike classical Convolutional Neural Networks (CNNs), the number of parameters of the filter is linear to the dimensionality of data. It constrains the scalability of the proposed filter. Second, the learned

graph signals ( and ) are vectors. It means that each vertex of users or items is represented by a scalar feature. However, a vector for every user and item is necessary to model the deep and complex connections between users and items.

Figure 3. The feed-forward procedure of SpectralCF. The function denotes the spectral convolution operation shown in Eq. (10).

The first limitation can be overcome by using a polynomial approximation. We first demonstrate that the set of all convolution filters is equal to the set of finite-order polynomials .

Proposition 3.2.

is equal to .

Proof.

Let us consider an instance . Then, . So, . Now, consider a convolution filter . Then, there must exist a polynomial function

that interpolates through all pairs

for . The maximum degree of such a polynomial is at most as there are maximum points to interpolate. Therefore, . ∎

Now, we can approximate the convolution filters by using first polynomials as the following:

(7)

In this way, the learning complexity of the filter becomes , where is a hyper-parameter, and independent from the number vertices. Specially, we limit the order of the polynomial, , to 1 in order to avoid over-fitting. By substituting Eq. (7) into Eq. (6), we have:

(8)

Furthermore, it is beneficial to further decrease the number of parameters by setting . As a result, Eq. (8) becomes:

(9)

where is a scalar.

For the second limitation, one can generalize the graph signals ( and ) to -dimensional graph signals: and . Hence, Eq. (9) becomes . To take one step further, we generalize the filter parameter to a matrix of filter parameters with input channels and filters. As a result, our final spectral convolution operation is shown as the following:

(10)

where and denote convolution results learned with filters from the spectral domain for users and items, respectively;

denotes the logistic sigmoid function.

In fact, Eq. (10) is a general version of Eq. (9) as it is equivalent to perform Eq. (9) in input channels with filters. Hereafter, the proposed convolution operation as shown in Eq. (10) is denoted as a function , which is parameterized by and .

3.4. Multi-layer Model

Given user vectors and item vectors , new graph singals ( and ) in Eq. (10) are convolution results learned from the spectral domain with a parameter matrix . As in classical CNNs, one can regard Eq. (10) as a propagation rule to build a deep neural feed-forward network based model, which we refer as Spectral Collaborative Filtering (SpectralCF).

Similar to word embedding techniques, we first randomly initialize user vectors and item vectors . Taking and as inputs, a layered deep spectralCF can be formulated as:

(11)

where is a matrix of filter parameters for the layer; and denote the convolution filtering results of the layer.

In order to utilize features from all layers of SpectralCF, we further concatenate them into our final latent factors of users and items as:

(12)

where and .

In terms of the loss function, the conventional BPR loss suggested in

(Rendle et al., 2009) is employed. BPR is a pair-wise loss to address the implicit data for recommendations. Unlike point-wise based methods (Koren, 2008), BPR learns a triple , where item is liked/clicked/viewed by user and item is not. By maximizing the preference difference between and , BPR assumes that the user prefers item over the unobserved item . In particular, given a user matrix and an item matrix as shown in Eq. (12), the loss function of SpectralCF is given as:

(13)

where and denote and column of and , respectively; represents the weight on the regularization terms. The training data is generated as:

(14)

3.5. Optimization and Prediction

At last, RMSprop

(Tieleman and Hinton, 2012) is used to minimize the loss function. The RMSprop is an adaptive version of gradient descent which adaptively controls the step size with respect to the absolute value of the gradient. It is done by scaling the updated value of each weight by a running average of its gradient norm.

As shown in Algorithm 1, for a batch of randomly sampled triple

, we update parameters in each epoch using the gradients of the loss function. After the training process, with optimized

, and , we derive the user ’s preference over item as . The final item recommendation for a user is given according to the ranking criterion as Eq. (15).

(15)

4. Experiments

Input: Training set: , number of epochs , batch size , number of layers , dimension of latent factors , number of filters , regularization term , learning rate , laplacian matrix and its corresponding eigenvectors and eigenvalues .
Output: Model’s parameter set: }.
1 Randomly initialize and

from a Gaussian distribution

;
2 for  do
3        Generate the batch of size by uniformly sampling from , and ;
4        for  do
5               Calculate and by using Eq. (10);
6        end for
7       Concatenate into and into ;
8       Estimate gradients by back propagation;
9        Update according to the procedure of RMSprop optimization (Tieleman and Hinton, 2012);
10 end for
return .
Algorithm 1 SpectralCF

As discussed in the introduction section, leveraging the connectivity information in a user-item bipartite graph is essentially important for an effective recommendation model. In this section, we argue that, directly learning from the spectral domain, the proposed SpectraCF can reveal the rich information of graph structures existing in the spectral domain for making better recommendations. One may ask the following research questions:

  • [leftmargin=0cm]

  • RQ1: How much does SpectralCF benefit from the connectivity information learned from the spectral domain?

  • RQ2: Does SpectralCF learn from the spectral domain in an effective way?

  • RQ3: Compared with traditional methods, can SpectralCF better counter the cold-start problem?

In this section, in order to answer the questions above, we conduct experiments to compare SpectralCF with state-of-the-art models.

4.1. Comparative Methods

To validate the effectiveness of SpectralCF, we compare it with six state-of-the-art models. The comparative models can be categorized into two groups: (1) CF-based Models: To answer RQ1, we compare SpectralCF with four state-of-the-art CF-based methods (ItemKNN, BPR, eALS and NCF) which ignore the information in the spectral domain; (2) Graph-based Models: For RQ2, we are interested in how effectively does SpetralCF learn the connectivity information from the spectral domain. We therefore compare SpectralCF with two graph-based models: GNMF and GCMC. Although the two models are also CF-based, we term them as graph-based models since they learn the structural information from a bipartite graph. These two groups of comparative models are summarized below:

  • ItemKNN (Sarwar et al., 2001): ItemKNN is a standard neighbor-based collaborative filtering method. The model finds similar items for a user based on their similarities.

  • BPR (Rendle et al., 2009): We use Bayesian Personalized Ranking based Matrix Factorization. BPR introduces a pair-wise loss into the Matrix Factorization to be optimized for ranking (Gantner et al., 2011).

  • eALS (He et al., 2016): This is a state-of-the-art matrix factorization based method for item recommendation. This model takes all unobserved interactions as negative instances and weighting them non-uniformly by the item popularity.

  • NCF (He et al., 2017): Neural Collaborative F

    iltering fuses matrix factorization and Multi-Layer Perceptron (MLP) to learn from user-item interactions. The MLP endows NCF with the ability of modelling non-linearities between users and items.

  • GNMF (Cai et al., 2008): Graph regularized Non-negative Matrix Facto-rization considers the graph structures by seeking a matrix factorization with a graph-based regularization.

  • GCMC (Berg et al., 2017): Graph Convolutional Matrix Completion utilizes a graph auto-encoder to learn the connectivity information of a bipartite interaction graph for latent factors of users and items.

Please note that, GNMF and GCMC are originally designed for explicit datasets. For a fair comparison, we follow the setting of (Hu et al., 2008) to adapt them for implicit data.

4.2. Datasets

Figure 4. Effects of hyper-parameter in terms of Recall@20 and MAP@20 in the dataset of MovieLens-1M.
Hyper-parameters
Values 3 0.001 0.001
Table 1. The hyper-parameter setting of SpectralCF.

We test our method as well as comparative models on three publicly available datasets333MovieLens-1M and HetRec are available at https://grouplens.org/datasets/; and Amazon Instant Video can be found at http://jmcauley.ucsd.edu/data/amazon/:

  • MovieLens-1M (Harper and Konstan, 2016): This movie rating dataset has been widely used to evaluate collaborative filtering algorithms. We used the version containing 1,000,209 ratings from 6,040 users for 3,900 movies. While it is a dataset with explicit feedbacks, we follow the convention (He et al., 2017) that transforms it into implicit data, where each entry is marked as 0 or 1 indicating whether the user has rated the item. After transforming, we retain a dataset of density.

  • HetRec (Cantador et al., 2011): This dataset has been released by the Second International Workshop on Information Heterogeneity and Fusion in Recommender Systems444http://ir.ii.uam.es/hetrec2011/. It is an extension of MovieLens-10M dataset and contains 855,598 ratings, 2,113 users and 10,197 movies. After converting it into implicit data as MovieLens-1M, we obtain a dataset of density.

  • Amazon Instant Video (McAuley et al., 2015): The dataset consists of 426,922 users, 23,965 videos and 583,933 ratings from Amazon.com. Similarly, we transformed it into implicit data and removed users with less than 5 interactions. As a result, a dataset of 0.12% density is obtained.

4.3. Experimental Setting

Ideally, a recommendation model should not only be able to retrieve all relevant items out of all items but also provide a rank for each user where relevant items are expected to be ranked in the top. Therefore, in our experiments, we use Recall@M and MAP@M to evaluate the performance of the top-M recommendations. Recall@M is employed to measure the fraction of relevant items retrieved out of all relevant items. MAP@M is used for evaluating the ranking performance of RS. The Recall@M for each user is then defined as:

(16)

The final results reported are average recall over all users.

For each dataset, we randomly select 80% items associated with each user to constitute the training set and use all the remaining as the test set. For each evaluation scenario, we repeat the evaluation five times with different randomly selected training sets and the average performance is reported in the following sections.

We use a validation set from the training set of each dataset to find the optimal hyper-parameters of comparative methods introduced in the Section 4.1

. For ItemKNN, we employ the cosine distance to measure item similarities. The dimensions of latent factors for BPR, eALS and GNMF are searched from {8,16,32,64,128} via the validation set. The hyperparameter

of eALS is selected from 0.001 to 0.04. Since the architecture of a multi-layer perceptron (MLP) is difficult to optimize, we follow the suggestion from the original paper (He et al., 2017) to employ a three-layer MLP with the shape of for NCF. The dropout rate of nodes for GCMC is searched from {0.3,0.4,0.5,0.6,0.7,0.8}. Our SpectralCF has one essential hyper-parameter: . Figure 4 shows how the performances of SpectralCF vary as is set from 1 to 5 on the validation set of MovieLens-1M. As we can see, in terms of Recall@20 and MAP@20, SpectralCF reaches its best performances when is fixed as 3. Other hyper-parameters of SpectralCF are empirically set and summarized in Table 1, where denotes the learning rate of RMSprop. Our models are implemented in TensorFlow (Abadi et al., 2016).

4.4. Experimental Results (RQ1 and RQ2)

(a) MovieLens-1M
(b) HetRec
(c) Amazon Instant Video
Figure 5.

Performance comparison in terms of recall@M with M varied from 20 to 100. Errors bars are 1-standard deviation.

(a) MovieLens-1M
(b) HetRec
(c) Amazon Instant Video
Figure 6. Performance comparison in terms of MAP@M with M varied from 20 to 100. Errors bars are 1-standard deviation.

In Figure 5, we compare SpectralCF with four CF-based models and two graph-based models in terms of Recall@M on all three datasets. Overall, when is varied from 20 to 100, SpectralCF consistently yields the best performance across all cases. Among CF-based comparative models, ItemKNN gives the worst performances in all three datasets, indicating the necessity of modeling users’ personalized preferences rather than just recommending similar items to users. For graph-based models (GNMF and GCMC), they generally underperform CF-based models such as BPR and NCF. The unsatisfying performance of GNMF shows that adding a graph-based regularization is not sufficient to capture complex structures of graphs. Though GCMC directly performs on a user-item bipartite graph, each vertex in the graph is only allowed to learn from its neighbors. This constrains its ability of capturing global structures in the graph. Among all comparative models, benefiting from its capability of modeling non-linear relationships between users and items, NCF beats all other models and becomes the strongest one. However, none of models above are able to directly perform in the spectral domain. They lose the rich information in the domain and as a result, SpectralCF greatly outperforms NCF by 16.1%, 16.2% and 28.0% in the dataset of MovieLen-1M, HetRec and Amazon Instant Video, respectively.

In Figure 6, we compare SpectralCF with all comparative models in terms of MAP@M. Again, when is in a range from 20 to 100, SpectralCF always yields the best performance. Neighbor-based ItemKNN performs the worst among all models. It further shows the advantages of modeling users’ personalized preferences. Compared with NCF and BPR, graph-based models (GNMF and GCMC) again fail to show convincing ranking performances measured by MAP@M. For CF-based models, while NCF beats other CF-based models in the dataset of HetRec, BPR shows itself as a strong model for ranking, owing to its pairwise ranking loss. It slightly outperforms NCF on average in the datasets of MovieLens-1M and Amazon Instant Video. However, SpectralCF improves BPR by 15.9%, 64.9% and 47.5% in the dataset of MovieLen-1M, HetRec and Amazon Instant Video, respectively.

Overall, as shown in Figure 5 and 6, not surprisingly, the performances of all models decline as the dataset becomes sparse. However, SpectralCF always outperforms all comparative models regardless of the sparsities of the datasets. By comparing spectralCF with traditional CF-based models, we demonstrate that the rich information of connectivity existing in the spectral domain assists SpectralCF in learning better latent factors of users and items. By comparing SpectralCF with graph-based models, we show that SpectralCF can effectively learn from the spectral domain.

4.5. Quality of Recommendations for Cold-start Users (RQ3)

To answer RQ3, in this section, we conduct an experiment to investigate the quality of recommendations made by SpectralCF for cold-start users. To this end, in the dataset of MovieLens-1M, we build training sets with different degrees of sparsity by varying the number of items associated with each user, denoted as , from one to five. All the remaining items associated with users are used as the test set. We compare SpectralCF with BPR, which is widely known and also shown as a strong ranking performer in Figure 6. The test results are reported in the Table 2.

In Table 2, it is shown that, suffering from the cold-start problem, the performances of BPR and SpectralCF inevitably degrade. However, regardless of the number of items associated with users, SpectralCF consistently outperforms BPR in terms of Recall@20 and MAP@20. On average, SpectralCF improves BPR by 36.8% and 33.8% in Recall@20 and MAP@20, respectively. Hence, it is demonstrated that compared with BPR, spectralCF can better handle cold-start users and provide more reliable recommendations.

5. Related Works

P 1 2 3 4 5
BPR 0.021 (0.003) 0.029 (0.004) 0.031 (0.003) 0.034 (0.004) 0.038 (0.003)
Recall @20 SpectralCF 0.031 (0.003) 0.039 (0.003) 0.042 (0.002) 0.045 (0.003) 0.051 (0.003)
Improve- ment 47.6% 34.5% 35.5% 32.4% 34.2%
BPR 0.014 (0.002) 0.017 (0.002) 0.021 (0.002) 0.024 (0.003) 0.027 (0.003)
MAP @20 SpectralCF 0.019 (0.002) 0.024 (0.002) 0.028 (0.003) 0.031 (0.003) 0.035 (0.002)
Improve- ment 35.7% 41.2% 33.3% 29.2% 29.6%
Table 2. Performance Comparison in terms of Recall@20 and MAP@20 in the sparse training sets. In the dataset of MovieLens-1M, we vary the number of items associated with each users, denoted as , from 1 to 5. The average results are reported and the best results are in bold. The standard deviation is shown in parentheses.

There are two categories of studies related to our work: deep learning based RS and graph-based RS. In this section, we will first briefly review existing works in the area of deep RS. Then, we focus on presenting recent works on graph-based RS. Despite all these approaches, SpectralCF is the first model to directly learn latent factors of users and items from the

spectral domains of user-item bipartite graphs.

5.1. Deep Recommender Systems

One of the early works utilizing deep learning for RS builds a Restricted Boltzmann Machines (RBM) based method to model users using their rating preferences

(Salakhutdinov et al., 2007). Although the method is still a relatively shallow model, it slightly outperforms Matrix Factorization technique and shows the promising future for deep recommender systems. In (Wang et al., 2017d), a generative model and a discriminative model are employed to play a minimax game. The two models are iteratively optimized and achieve promising results for the item recommendation problem. Inspired by (Salakhutdinov et al., 2007), (Zheng et al., 2016) proposed a CF Neural Autoregressive Distribution Estimator (CF-NADE) model for collaborative filtering tasks. CF-NADE shares parameters between different ratings. (He et al., 2017)

presents to utilize a Multilayer Perceptron (MLP) to model user-item interactions.

A number of researchers proposed to build a hybrid recommender systems to counter the sparsity problem. (Wang and Wang, 2014)

introduce Convolutional Neural Networks (CNN) and Deep Belief Network (DBN) to assist representation learning for music data. As such, their model is able to extract latent factors of songs without ratings while CF based techniques like MF are unable to handle these songs. These approaches above pre-train embeddings of users and items with matrix factorization and utilize deep models to fine-tune the learned item features based on item content. In

(Elkahky et al., 2015) and (Wang et al., 2017a), multi-view deep models are built to utilize item information from more than one domain. (Kim et al., 2016) integrates a CNN with PMF to analyze documents associated with items to predict users’ future explicit ratings. (Zheng et al., 2017b) leverage two parallel neural networks to jointly model latent factors of users and items. To incorporate visual signals into RS, (Wang et al., 2017b)

propose CNN-based models to incorporate visual signals into RS. They make use of visual features extracted from product images using deep networks to enhance the performance of RS.

(Zhang et al., 2016) investigates how to leverage the multi-view information to improve the quality of recommender systems. (Cheng et al., 2016) jointly trains wide linear models and deep neural networks for video recommendations. (Wang et al., 2016) and (Zheng et al., 2017a) utilize RNN to consider word orders and extract complex semantics for recommendations. (Wang et al., 2017c) applies an attention mechanism on a sequence of models to adaptively capture the change of criteria of editors. (Zheng et al., 2018)

leverages an attentional model to learn adaptive user embeddings. A survey on the deep learning based RS with more works on this topic can be found in

(Zhang et al., 2017).

5.2. Graph-based Recommender Systems

In order to learn latent factors of users and items from graphs, a number of researchers have proposed graph-based RS. (Zhou et al., 2008)

develops a semi-supervised learning model on graphs for document recommendation. The model combines multiple graphs in order to measure item similarities. In

(Yuan et al., 2014), they propose to model the check-in behaviors of users and a graph-based preference propagation algorithm for point of interest recommendation. The proposed solution exploits both the geographical and temporal influences in an integrated manner. (Guan et al., 2009) addresses the problem of personalized tag recommendation by modeling it as a ”query and ranking” problem. Inspired by the recent success of graph/node embedding methods, (Berg et al., 2017) proposes a graph convolution network based model for recommendations. In (Berg et al., 2017), a graph auto-encoder learns the structural information of a graph for latent factors of users and items. (Cai et al., 2008) adds graph-based regularizations into the matrix factorization model to learn graph structures. Graph-regularized methods are developed for the problemm of matrix completion in (Rao et al., 2015). (Monti et al., 2017)

combines a convolutional neural network and a recurrent neural network to model the dynamic rating generation process. Although this work also considers the

spectral domain, they learn from a graph constructed from side information, such as genres or actors for movies. In contrast, our method learns directly from user-item bipartite graphs and does not require the side information. Thus, this work is not comparable to our method.

Additionally, some scholars have proposed to incorporate the heterogeneous information on a graph for recommendations. (Jamali and Lakshmanan, 2013) suggests a general latent factor model for entities in a graph. (Yu et al., 2013) introduces a recommendation model for implicit data by taking advantage of different item similarity semantics in the graph. (Shi et al., 2015) introduces a semantic path based personalized recommendation method to predict the rating scores of users on items.

However, all works above are different from ours because they fail to consider the rich information in the spectral domains of user-item bipartite graphs. Also, our study focuses on learning from the implicit feedbacks, and leaves incorporating the heterogeneous information in a graph and the item content for future works.

6. Conclusions

It is shown that the rich information of connectivity existing in the spectral domain of a bipartite graph is helpful for discovering deep connections between users and items. In this paper, we introduce a new spectral convolution operation to directly learn latent factors of users and items from the spectral domain

. Furthermore, with the proposed operation, we build a deep feed-forward neural network based recommendation model, named Spectral Collaborative Filtering (SpectralCF). Due to the rich information of connectivity existing in the

spectral domain, compared with previous works, SpectralCF is capable of discovering deep connections between users and items and therefore, alleviates the cold-start problem for CF. To the best of our knowledge, SpectralCF is the first CF-based method directly learning from the spectral domains of user-item bipartite graphs. We believe that it shows the potential of conducting CF in the spectral domain, and will encourage future works in this direction.

In comparison with four state-of-the-art CF-based and two graph-based models, SpectralCF achieved 20.1% and 42.6% improvements averaging on three standard datasets in terms of Recall@M and MAP@M, respectively.

Additionally, in the experiments, by varying the number of items associated with each user from 1 to 5, we build training sets with different degrees of sparsity to investigate the quality of recommendations made by SpectralCF for cold-start users. By comparing SpectralCF with BPR, on average, SpectralCF improves BPR by 36.8% and 33.8% in Recall@20 and MAP@20, respectively. It is validated that SpectralCF can effectively ameliorate the cold-start problem.

Acknowledgements.
This work is supported in part by NSF through grants IIS-1526499, IIS-1763325, and CNS-1626432, and NSFC 61672313. This work is also partially supported by NSF through grant IIS-1763365 and by FSU through the startup package and FYAP award.

References

  • (1)
  • Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). Savannah, Georgia, USA.
  • Berg et al. (2017) Rianne van den Berg, Thomas N Kipf, and Max Welling. 2017. Graph Convolutional Matrix Completion. arXiv preprint arXiv:1706.02263 (2017).
  • Cai et al. (2008) Deng Cai, Xiaofei He, Xiaoyun Wu, and Jiawei Han. 2008. Non-negative matrix factorization on manifold. In Data Mining, 2008. ICDM’08. Eighth IEEE International Conference on. IEEE, 63–72.
  • Cantador et al. (2011) Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. 2011. 2nd Workshop on Information Heterogeneity and Fusion in Recommender Systems (HetRec 2011). In Proceedings of the 5th ACM conference on Recommender systems (RecSys 2011). ACM, New York, NY, USA.
  • Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 7–10.
  • Defferrard et al. (2016) Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems. 3844–3852.
  • Elkahky et al. (2015) Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. 2015. A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 278–288.
  • Gantner et al. (2011) Zeno Gantner, Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2011. MyMediaLite: A Free Recommender System Library. In 5th ACM International Conference on Recommender Systems (RecSys 2011).
  • Guan et al. (2009) Ziyu Guan, Jiajun Bu, Qiaozhu Mei, Chun Chen, and Can Wang. 2009. Personalized tag recommendation using graph-based ranking on multi-type interrelated objects. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. ACM, 540–547.
  • Harper and Konstan (2016) F Maxwell Harper and Joseph A Konstan. 2016. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4 (2016), 19.
  • He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017. 173–182. https://doi.org/10.1145/3038912.3052569
  • He et al. (2016) Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016. Fast matrix factorization for online recommendation with implicit feedback. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 549–558.
  • Hu et al. (2008) Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining. Ieee, 263–272.
  • Jamali and Ester (2009) Mohsen Jamali and Martin Ester. 2009. Trustwalker: a random walk model for combining trust-based and item-based recommendation. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 397–406.
  • Jamali and Lakshmanan (2013) Mohsen Jamali and Laks Lakshmanan. 2013. HeteroMF: recommendation in heterogeneous information networks using context dependent factor models. In Proceedings of the 22nd international conference on World Wide Web. ACM, 643–654.
  • Kim et al. (2016) Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. 2016. Convolutional matrix factorization for document context-aware recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 233–240.
  • Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
  • Koren (2008) Yehuda Koren. 2008. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 426–434.
  • Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer 8 (2009), 30–37.
  • McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based Recommendations on Styles and Substitutes. arXiv preprint arXiv:1506.04757 (2015).
  • Monti et al. (2017) Federico Monti, Michael Bronstein, and Xavier Bresson. 2017. Geometric matrix completion with recurrent multi-graph neural networks. In Advances in Neural Information Processing Systems. 3700–3710.
  • Rao et al. (2015) Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon. 2015. Collaborative filtering with graph information: Consistency and scalable methods. In Advances in neural information processing systems. 2107–2115.
  • Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In

    Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence

    . AUAI Press, 452–461.
  • Salakhutdinov et al. (2007) Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. 2007. Restricted Boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on Machine learning. ACM, 791–798.
  • Sarwar et al. (2001) Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. ACM, 285–295.
  • Shi et al. (2015) Chuan Shi, Zhiqiang Zhang, Ping Luo, Philip S Yu, Yading Yue, and Bin Wu. 2015. Semantic path based personalized recommendation on weighted heterogeneous information networks. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, 453–462.
  • Shuman et al. (2013) David I Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013.

    The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains.

    IEEE Signal Processing Magazine 30, 3 (2013), 83–98.
  • Spielman (2007) Daniel A Spielman. 2007. Spectral graph theory and its applications. In Foundations of Computer Science, 2007. FOCS’07. 48th Annual IEEE Symposium on. IEEE, 29–38.
  • Tieleman and Hinton (2012) Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning 4 (2012), 2.
  • Wang et al. (2017a) Fengjiao Wang, Yongzhi Qu, Lei Zheng, Chun-Ta Lu, and S Yu Philip. 2017a. Deep and Broad Learning on Content-Aware POI Recommendation. In Collaboration and Internet Computing (CIC), 2017 IEEE 3rd International Conference on. IEEE, 369–378.
  • Wang et al. (2016) Hao Wang, SHI Xingjian, and Dit-Yan Yeung. 2016.

    Collaborative recurrent autoencoder: Recommend while learning to fill in the blanks. In

    Advances in Neural Information Processing Systems. 415–423.
  • Wang et al. (2017d) Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017d. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 515–524.
  • Wang et al. (2017b) Suhang Wang, Yilin Wang, Jiliang Tang, Kai Shu, Suhas Ranganath, and Huan Liu. 2017b. What your images reveal: Exploiting visual contents for point-of-interest recommendation. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 391–400.
  • Wang and Wang (2014) Xinxi Wang and Ye Wang. 2014. Improving content-based and hybrid music recommendation using deep learning. In Proceedings of the ACM International Conference on Multimedia. ACM, 627–636.
  • Wang et al. (2017c) Xuejian Wang, Lantao Yu, Kan Ren, Guanyu Tao, Weinan Zhang, Yong Yu, and Jun Wang. 2017c. Dynamic attention deep model for article recommendation by learning human editors’ demonstration. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2051–2059.
  • Yu et al. (2013) Xiao Yu, Xiang Ren, Yizhou Sun, Bradley Sturt, Urvashi Khandelwal, Quanquan Gu, Brandon Norick, and Jiawei Han. 2013. Recommendation in heterogeneous information networks with implicit user feedback. In Proceedings of the 7th ACM conference on Recommender systems. ACM, 347–350.
  • Yuan et al. (2014) Quan Yuan, Gao Cong, and Aixin Sun. 2014. Graph-based point-of-interest recommendation with geographical and temporal influences. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 659–668.
  • Zhang et al. (2016) Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016. Collaborative knowledge base embedding for recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 353–362.
  • Zhang et al. (2017) Shuai Zhang, Lina Yao, and Aixin Sun. 2017. Deep learning based recommender system: A survey and new perspectives. arXiv preprint arXiv:1707.07435 (2017).
  • Zheng et al. (2017a) Lei Zheng, Bokai Cao, Vahid Noroozi, S Yu Philip, and Nianzu Ma. 2017a. Hierarchical collaborative embedding for context-aware recommendations. In Big Data (Big Data), 2017 IEEE International Conference on. IEEE, 867–876.
  • Zheng et al. (2018) Lei Zheng, Chun-Ta Lu, Lifang He, Sihong Xie, Vahid Noroozi, He Huang, and Philip S Yu. 2018. MARS: Memory Attention-Aware Recommender System. arXiv preprint arXiv:1805.07037 (2018).
  • Zheng et al. (2017b) Lei Zheng, Vahid Noroozi, and Philip S Yu. 2017b. Joint Deep Modeling of Users and Items Using Reviews for Recommendation. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, 425–434.
  • Zheng et al. (2016) Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. 2016. A neural autoregressive approach to collaborative filtering. arXiv preprint arXiv:1605.09477 (2016).
  • Zhou et al. (2008) Ding Zhou, Shenghuo Zhu, Kai Yu, Xiaodan Song, Belle L Tseng, Hongyuan Zha, and C Lee Giles. 2008. Learning multiple graphs for document recommendations. In Proceedings of the 17th international conference on World Wide Web. ACM, 141–150.