Online news services such as Google News111https://news.google.com/ and MSN News222https://www.msn.com/en-us/news which collect and aggregate news from many sources have become popular platforms for news reading [Das et al.2007, Lavie et al.2010]. Tens of thousands of news are generated everyday, and it is impossible for users to read all these news articles due to time limit [Phelan et al.2011]. Therefore, personalized news recommendation is very important for online news platforms to help users find their interested news and alleviate information overload [IJntema et al.2010, De Francisci Morales et al.2012].
Many deep learning based news recommendation methods have been proposed[Okura et al.2017, Wang et al.2018]. A core problem in these methods is how to learn representations of news and users. Existing methods usually learn these representations based on single news information. For example, Okura et al. okura2017embedding proposed to learn news representations from the body of news articles via auto-encoders, and then learn users representations from news representations by applying GRU to their browsed news. Wang et al. wang2018dkn proposed to learn news representations via a knowledge-aware CNN from the titles of news articles, and then learn user representations from news representations based on the similarity between candidate news and each browsed news. The single kind of news information may be insufficient for learning accurate news and user representations, and may limit the performance of news recommendation.
Our work is motivated by the following observations. First, a news article usually contains different kinds of information, such as title, body and topic category, which are all useful for representing news. For example, in Fig. 1 the title of the first news indicates that it is about a deal between a baseball star and a team, and the body can provide more detailed information of this deal. In addition, the topic category of this news is also informative, since if a user clicks this news and many other news articles with the same category, then we can infer she is very interested in sports. Thus, incorporating different kinds of news information has the potential to improve news and user representations for news recommendation. Second, different kinds of news information have different characteristics. For example, the titles are usually very short and concise, while the bodies are much longer and more detailed. The topic categories are usually labels with very few words. Thus, different kinds of news information should be processed differently. Third, different news information may have different informativeness for different news. For example, the title of the first news in Fig. 1 is precise and important for representing this news, while the title of the second news is short, vague, and less informative. Besides, different words in the same news may have different importance. For example, in Fig. 1, “outfileder” is more important than “familiar” for representing the first news. In addition, different news browsed by the same user may also have different informativeness for learning user representations. For example, if a user browses a news of “Super Bowl” which is extremely popular and a news of “International Mathematical Olympiad” which is less popular, then the second news may be more informative for inferring her interests than the first one.
In this paper, we propose a neural news recommendation approach with attentive multi-view learning (NAML) to learn informative news and user representations by exploiting different types of news information. Our approach contains two core components, i.e., a news encoder and a user encoder. In the news encoder, we propose an attentive multi-view learning framework to learn unified news representations from titles, bodies and categories by incorporating them as different views of news. In addition, since different words and views may have different informativeness for news representation, we apply word-level and view-level attention networks to select important words and views for learning informative news representations. In the user encoder, we learn the representations of users from the representations of their clicked news. Since different news may also have different informativeness for user representations, we apply attention mechanism to news encoder to select informative news for user representation learning. We conduct extensive experiments on a real-world dataset. The results show our approach can effectively improve the performance of news recommendation.
2 Related Work
News recommendation is an important task in natural language processing and data mining fields, and has wide applications[Zheng et al.2018, Wu et al.2019a]. Learning accurate news and user representations is critical for news recommendation. Many existing news recommendation methods rely on manual feature engineering for news and user representation learning [Liu et al.2010, Son et al.2013, Bansal et al.2015]. For example, Liu et al. liu2010personalized proposed to use topic categories and interest features generated by a Bayesian model as news representations. Son et al. son2013location proposed an Explicit Localized Semantic Analysis (ELSA) model for location-based news recommendation. They proposed to extract topic and location features from Wikipedia pages as news representations. Lian et al. lian2018towards proposed a deep fusion model (DMF) to learn news representations from various handcrafted features such as title length and news categories. However, these methods rely on manual feature engineering, which needs a large amount of domain knowledge and effort to craft. In addition, these methods cannot capture contexts and orders of words in news, which are important for learning news and user representations.
. For example, Okura et al. okura2017embedding proposed to learn news representations from news bodies using denoising autoencoders. In addition, they proposed to learn user representations from their browsed news using GRU network. Wang et al. wang2018dkn proposed to learn news representations from news titles via a knowledge-aware CNN network which can incorporate the information in knowledge graphs. However, these methods can only exploit a single kind of news information, which may be insufficient for learning accurate representations of news and users. Different from these methods, our approach can learn representations of news and users by incorporating different kinds of news information such as title, body and topic category via an attentive multi-view learning framework. Extensive experiments on real-world dataset validate our approach can learn better news and user representations, and achieve better performance on news recommendation than existing methods.
3 Our Approach
In this section, we introduce our NAML approach for neural news recommendation. There are three major modules in our approach, i.e., a news encoder with attentive multi-view learning to learn representations of news, a user encoder with attention mechanism to learn representations of users from their browsed news, and a click predictor
to predict the probability of a user browsing a candidate news article. The architecture of our approach is shown in Fig.2.
3.1 News Encoder
The news encoder module is used to learn representations of news from different kinds of news information, such as titles, bodies and topic categories. Since different kinds of news information have different characteristics, instead of simply merging them into a long text for news representation, we propose an attentive multi-view learning framework to learn unified news representations by regarding each kind of news information as a specific view of news. As shown in Fig. 2, there are four major components in news encoder.
The first component is title encoder, which is used to learn news representations from their titles. There are three layers in the title encoder
. The first layer is word embedding, which is used to convert a news title from a word sequence into a sequence of low-dimensional semantic vectors. Denote the word sequence of a news title as, where is the length of this title. It is converted into a sequence of word vectors via a word embedding look-up table , where and are vocabulary size and word embedding dimension respectively.
The second layer is a convolutional neural network (CNN)[Kim2014]. Local contexts of words in news titles are important for learning their representations. For example, in the news title “Xbox One On Sale This Week”, the local contexts of “One” such as “Xbox” and “On Sale” are useful for understanding that it belongs to a game console name. Thus, we use CNNs to learn contextual word representations by capturing their local contexts. Denote the contextual representation of the -th word as , which is calculated by:
where is the concatenation of word embeddings from position to . and are the kernel and bias parameters of the CNN filters, is the number of CNN filters and
is their window size. ReLU[Glorot et al.2011]
is the non-linear activation function. The output of this layer is a sequence of contextual word representations, i.e.,.
The third layer is a word-level attention network [Wu et al.2019b]. Different words in the same news title usually have different informativeness for learning news representations. For example, in the news title “Rockets End 2018 With A Win” the word “Rockets” is more informative than “With” in representing this news. Thus, recognizing the important words in different new titles has the potential to learn more informative news representations. We propose to use a word-level attention network to select important words within the context of each news title. Denote the attention weight of the word in a news title as , which is formulated as:
where and are the projection parameters, denotes the attention query vector.
The final representation of a news title is the summation of the contextual representations of its words weighted by their attention weights, i.e., .
The second component in news encoder module is body encoder, which is used to learn news representations from their bodies. Similar with the title encoder, there are also three layers in the body encoder.
The first layer is word embedding, which is shared with the title encoder. Denote the word sequence of a news body as , where is the length of the body. Through the word embedding layer, it is transformed into a sequence of word vectors . The second layer is a CNN network, which takes the word vector sequence as input, and learns the contextual word representations by capturing the local contexts. The third layer is an attention network. Different words in the same news body may have different informativeness for news representations. For example, in Fig. 1 the word “outfielder” is more informative than “familiar” in representing this news. Thus, we apply an attention network to the body encoder to select important words in the context of each news body for learning more informative news representations. Denote the attention weight of the word in a news body as , which is calculated by:
where and are projection parameters, and is the attention query vector. The final representation of a news body is the summation of the contextual word representations weighted by their attention weights, i.e., .
The third component in the news encoder module is category encoder, which is used to learn news representations from their categories. On many online news platforms such as MSN News, news articles are usually labeled with topic categories (e.g., “sports” and “entertainment”) and subcategories (e.g., “basketball_nba” and “gaming”) for targeting user interests which contain important information of these news. For example, if a user clicked many news articles with the category of “sports”, then we can infer that this user may be interested in sports news. Thus, we propose to incorporate both the category and subcategory information for news representation learning. The inputs of the category encoder are the ID of the category and the ID of the subcategory . There are two layers in the category encoder. The first layer is a category ID embedding layer. It converts the discrete IDs of categories and subcategories into low-dimensional dense representations (denoted as and respectively). The second layer is a dense layer. It is used to learn the hidden category representations by transforming the category embeddings as:
where , , and are parameters in dense layers.
The fourth component in the news encoder module is attentive pooling. Different kinds of news information may have different informativeness for learning the representations of different news. For example, in Fig. 1, the title of the first news is precise and contains rich information. Thus, it should have a high weight in representing this news. However, the title of the second news is short and vague. Thus, the body and the topic category should have higher weights than the title for representing this news. Motivated by these observations, we propose a view-level attention network to model the informativeness of different kinds of news information for learning news representations. Denote the attention weights of title, body, category and subcategory as , , and respectively. The attention weight of the title view is calculated by:
where and are projection parameters, is the attention query vector. The attention weights of other news information such as body, category and subcategory can be computed in a similar way.
The final unified news representations learned by the news encoder module is the summation of the news representations from different views weighted by their attention weights:
In our NAML approach the news encoder module is used to learn the representations of both historical news browsed by users and the candidate news to be recommended.
3.2 User Encoder
The user encoder module in our approach is used to learn the representations of users from the representations of their browsed news, as shown in Fig. 2
. Different news browsed by the same user have different informativeness for representing this user. For example, the news “10 best NBA moments” is very informative for modeling user preferences since it is usually browsed by users who are interested in sports, but the news “The Weather Next Week” is less informative since it is clicked by massive users. Therefore, inuser encoder module we apply a news attention network to learn more informative user representations by selecting important news. We denote the attention weight of the news browsed by a user as , which is calculated as follows:
where , and are the parameters in the news attention network. The final representation of a user is the summation of the representations of news browsed by this user weighted by their attention weights: , where is the number of browsed news.
3.3 Click Predictor
The click predictor module is used to predict the probability of a user browsing a candidate news based on their representations. Denote the representation of a candidate news as and the representation of user as . Following [Okura et al.2017], the click probability score is calculated by the inner product of the representation vectors of user and the candidate news , i.e., We also explored other probability computation methods such as multi-layer neural networks [Wang et al.2018]. We find that the inner product is not only the one with the best time efficiency but also the one with the best performance, which is consistent with [Okura et al.2017].
3.4 Model Training
Motivated by [Huang et al.2013] and [Zhai et al.2016], we propose to use negative sampling techniques for model training. For each news browsed by a user which is regarded as a positive sample, we randomly sample news which are presented in the same session but are not clicked by this user as negative samples. We then jointly predict the click probability scores of the positive news and the negative news . In this way, we formulate the news click prediction problem as a pseudo -way classification task. We normalize these click probability scores using softmax to compute the posterior click probability of a positive sample as follows:
where is the click probability score of the -th positive news, and is the click probability score of the -th negative news in the same session with the
-th positive news. The loss functionin our approach for model training is the negative log-likelihood of all positive samples, which can be formulated as follows:
where is the set of the positive training samples.
4.1 Datasets and Experimental Settings
Our experiments were conducted on a real-world dataset, which was constructed by randomly sampling user logs from MSN News333https://www.msn.com/en-us/news in one month, i.e., from December 13, 2018 to January 12, 2019. The detailed statistics of the dataset are shown in Table 1. We use the logs in the last week as the test set, and the rest logs are used for training. In addition, we randomly sampled 10% of training samples for validation.
|# users||10,000||avg. # words per title||11.29|
|# news||42,255||avg. # words per body||730.72|
|# impressions||445,230||# positive samples||489,644|
|# samples||7,141,584||# negative samples||6,651,940|
In our experiments, the dimensions of word embeddings and category embeddings were set to 300 and 100 respectively. We used the pre-trained Glove embedding [Pennington et al.2014] in our approach. The number of CNN filters was set to 400, and the window size was 3. The dimension of the dense layers in the category view was set to 400. The sizes of attention queries was set to 200. The negative sampling ratio was set to 4. We applied 20% dropout [Srivastava et al.2014] to each layer in our approach to mitigate overfitting. Adam [Kingma and Ba2014]
was used as the optimization algorithm. The batch size was set to 100. These hyperparameters were selected according to the validation set. The metrics in our experiments include the average AUC, MRR, nDCG@5 and nDCG@10 scores over all impressions. We independently repeated each experiment for 10 times and reported the average performance.
4.2 Performance Evaluation
First, we will evaluate the performance of our approach by comparing it with several baseline methods. The methods to be compared include: (1) LibFM [Rendle2012], which is a state-of-the-art feature-based matrix factorization method; (2) CNN [Kim2014], applying CNN to the concatenation of news titles, bodies and categories to learn news representations; (3) DSSM [Huang et al.2013], a deep structured semantic model with word hashing via character trigram and multiple dense layers; (4) Wide&Deep [Cheng et al.2016], using the combination of a wide linear channel and a deep neural network channel; (5) DeepFM [Guo et al.2017], using a combination with factorization machines and neural networks; (6) DFM [Lian et al.2018], a deep fusion model by using combinations of dense layers with different depths; (7) DKN [Wang et al.2018], a deep news recommendation method based on knowledge-aware CNN; (8) NAML, our neural news recommendation approach with attentive multi-view learning. The experimental results of different methods are summarized in Table 2.
According to Table 2, We have several observations. First, the methods based on neural networks (e.g., CNN, DSSM and NAML) outperform traditional matrix factorization methods such as LibFM. This is probably because neural networks can learn better news and user representations than traditional matrix factorization methods.
Second, the methods using negative sampling (DSSM and NAML) outperform the methods without negative sampling (e.g., CNN, DFM and DKN). This is probably because the methods without negative sampling are trained on a balanced dataset, which is not consistent with real-word news recommendation scenarios. DSSM and our NAML approach can utilize the information from much more negative samples than other baseline methods, which is useful for learning more accurate news representations.
Third, our approach can consistently outperform other baseline methods. Since the characteristics of news titles, bodies and categories are usually divergent, simply concatenating the features extracted from these sources may be sub-optimal. Different from baseline methods, our approach uses a multi-view framework to incorporate different information as different views. In addition, different words, news and views may have different informativeness for representing news and users. Our approach can simultaneously select important words, news and views, which can build more informative news and user representations.
4.3 Effectiveness of Attentive Multi-View Learning
In this section, we conducted several experiments to validate the effectiveness of attentive multi-view learning framework in our model. First, we explore the effectiveness of multi-view framework in our NAML approach. The performance of NAML and its variants with different combinations of views is shown in Fig. 3(a). According to Fig. 3(a), we have several observations. First, the model with the body view achieves better performance than those with the title or category view only. This is intuitive because the bodies of news usually contain the original information of news, and can provide rich information for modeling news topics. Second, the title and category views are also informative for news recommendation. This is probably because titles usually have decisive influence on users’ reading behaviors. Thus, incorporating news titles is useful for modeling the characteristics of news and users. In addition, since categories of news are important clues of news topics, incorporating the category view is also useful for recommendation. Third, combine all three views can further improve the performance of our approach. These results validate the effectiveness of our multi-view framework.
Next, we conducted experiments to validate the effectiveness of the attention mechanism at word-, news- and view-level. The performance of NAML and its variant with different combinations of attention networks is shown in Fig. 3(b). According to Fig. 3(b), we have several observations. First, the word-level attention network can effectively improve the performance of our approach. This is probably because words are basic units in titles and bodies to convey their meanings, and different words usually have different informativeness for learning news representations. Our approach can recognize and highlight the important words, which is useful for learning more informative news representations. Second, the news-level attention is also important to improve the performance of our approach. This is probably because news browsed by the same user usually have different informativeness for learning the representations of this user, and recognizing important news is useful for learning high-quality user representations. Third, using view-level attention network can also improve the performance of our approach. Since the three views can also have different informativeness for modeling news and users, evaluating the importance of views may be useful for improving the performance of our approach. Fourth, combining all attention networks can further improve the performance of our approach. These results validate the effectiveness of attention mechanism in our approach.
4.4 Visualization of Attention Weights
In this section, we will visually explore the effectiveness of the attention mechanism in our approach. First, we want to visualize the view-level attention weights of the three views. The results are shown in Fig. 4. From Fig. 4(a), we find the average attention weights of the body view are higher than those of the title view. This may be because bodies usually convey the original meanings of news and contain richer information than titles. Thus, the attention weights of the body view is higher. Second, we find it is interesting that the category view gains the highest attention weights. This may be because categories are very important clues for inferring news topics, which are very informative for learning accurate news representations. The visualization results of distributions of the view-level attention weights are shown in Fig. 4(b). We find that the attention weights on the title and body views are small for many samples. This may be because some news titles and bodies are vague and uninformative for learning news representations. In these cases, categories are can provide rich complementary information for recommendation, which validate the effectiveness of our multi-view framework.
Then, we want to visually explore the effectiveness of the word- and news-level attention networks. For the word-level attention network, we only visualize the attention weights of the words in news titles since news bodies are usually too long. The visualization results of several clicked news from a randomly selected users are shown in Fig. 5. From Fig. 5, we find the attention network can effectively recognize important words within news articles. For example, the word “NBA” and “Rockets” are highlighted, since these words are very informative for representing news topics, while the word “December” is assigned a low attention weight since it is less informative for learning news topics. These results show that our approach can learn informative news representations by recognizing important words. From Fig. 5, we also find our approach can also effectively recognize important news of a user. For example, the news “Rockets set NBA 3 - point record , blast Wizards” gains high attention weight because it is very informative for modeling user preferences, while the news “Today in History : December 22” gains low attention weight, since it is not very informative for representing users. These results show that our model can effectively evaluate the different importance of words and news articles for learning informative news representations.
In this paper, we propose a neural news recommendation approach with attentive multi-view learning (NAML). The core of our approach is a news encoder and a user encoder. In the news encoder, we propose a multi-view learning framework to learn unified news representations by incorporating titles, bodies and categories as different views of news. We also apply attention mechanism to news encoder to select important words and views for learning informative news representations. In the user encoder, we learn representations of users from their browsed news, and apply a news attention network to select important news for learning informative user representations. Extensive experiments on real-world dataset show our approach can improve the performance of news recommendation and outperform many baseline methods.
The authors would like to thank Microsoft News for providing technical support and data in the experiments, and Jiun-Hung Chen (Microsoft News) and Ying Qiao (Microsoft News) for their support and discussions. This work was supported by the National Key Research and Development Program of China under Grant number 2018YFC1604002, the National Natural Science Foundation of China under Grant numbers U1836204, U1705261, U1636113, U1536201, and U1536207, and the Tsinghua University Initiative Scientific Research Program under Grant number 20181080368.
- [Bansal et al.2015] Trapit Bansal, Mrinal Das, and Chiranjib Bhattacharyya. Content driven user profiling for comment-worthy recommendations of news and blog articles. In RecSys., pages 195–202. ACM, 2015.
- [Cheng et al.2016] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In DLRS, pages 7–10, 2016.
- [Das et al.2007] Abhinandan S Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. Google news personalization: scalable online collaborative filtering. In WWW, pages 271–280. ACM, 2007.
- [De Francisci Morales et al.2012] Gianmarco De Francisci Morales, Aristides Gionis, and Claudio Lucchese. From chatter to headlines: harnessing the real-time web for personalized news recommendation. In WSDM, pages 153–162. ACM, 2012.
- [Glorot et al.2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, pages 315–323, 2011.
- [Guo et al.2017] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr prediction. In AAAI, pages 1725–1731. AAAI Press, 2017.
- [Huang et al.2013] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In CIKM, pages 2333–2338. ACM, 2013.
- [IJntema et al.2010] Wouter IJntema, Frank Goossen, Flavius Frasincar, and Frederik Hogenboom. Ontology-based news recommendation. In Proceedings of the 2010 EDBT/ICDT Workshops, page 16. ACM, 2010.
- [Khattar et al.2018] Dhruv Khattar, Vaibhav Kumar, Vasudeva Varma, and Manish Gupta. Weave& rec: A word embedding based 3-d convolutional network for news recommendation. In CIKM, pages 1855–1858. ACM, 2018.
- [Kim2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746–1751, 2014.
- [Kingma and Ba2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- [Kumar et al.2017] Vaibhav Kumar, Dhruv Khattar, Shashank Gupta, and Vasudeva Varma. Word semantics based 3-d convolutional neural networks for news recommendation. In 2017 IEEE International Conference on Data Mining Workshops, pages 761–764, 2017.
- [Lavie et al.2010] Talia Lavie, Michal Sela, Ilit Oppenheim, Ohad Inbar, and Joachim Meyer. User attitudes towards news content personalization. International journal of human-computer studies, 68(8):483–495, 2010.
- [Lian et al.2018] Jianxun Lian, Fuzheng Zhang, Xing Xie, and Guangzhong Sun. Towards better representation learning for personalized news recommendation: a multi-channel deep fusion approach. In IJCAI, pages 3805–3811, 2018.
- [Liu et al.2010] Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. Personalized news recommendation based on click behavior. In IUI, pages 31–40. ACM, 2010.
- [Okura et al.2017] Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. Embedding-based news recommendation for millions of users. In KDD, pages 1933–1942. ACM, 2017.
- [Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543, 2014.
- [Phelan et al.2011] Owen Phelan, Kevin McCarthy, Mike Bennett, and Barry Smyth. Terms of a feather: Content-based news recommendation and discovery using twitter. In ECIR, pages 448–459. Springer, 2011.
- [Rendle2012] Steffen Rendle. Factorization machines with libfm. TIST, 3(3):57, 2012.
- [Son et al.2013] Jeong-Woo Son, A Kim, Seong-Bae Park, et al. A location-based news article recommendation with explicit localized semantic analysis. In SIGIR, pages 293–302. ACM, 2013.
- [Srivastava et al.2014] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958, 2014.
- [Wang et al.2018] Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Dkn: Deep knowledge-aware network for news recommendation. In WWW, pages 1835–1844, 2018.
- [Wu et al.2019a] Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. Neural news recommendation with topic-aware news representation. In ACL, 2019.
[Wu et al.2019b]
Chuhan Wu, Fangzhao Wu, Junxin Liu, Shaojian He, Yongfeng Huang, and Xing Xie.
Neural demographic prediction using search query.In WSDM, pages 654–662. ACM, 2019.
- [Wu et al.2019c] Chuhan Wu, Fangzhao Wu, Junxin Liu, and Yongfeng Huang. Npa: Neural news recommendation with personalized attention. In KDD. ACM, 2019.
[Zhai et al.2016]
Shuangfei Zhai, Keng-hao Chang, Ruofei Zhang, and Zhongfei Mark Zhang.
Deepintent: Learning attentions for online advertising with recurrent neural networks.In KDD, pages 1295–1304. ACM, 2016.
[Zheng et al.2018]
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing
Xie, and Zhenhui Li.
Drn: A deep reinforcement learning framework for news recommendation.In WWW, pages 167–176, 2018.