In many e-commerce online scenarios, user profiles usually cannot be obtained so that session-based recommendation has become an important solution to address anonymous recommendation. Session-based recommender (SBR) system learns users’ preferences by mining sequential patterns from users’ chronological historical behavior without user profiles to predict users’ future interests in one session. Most traditional Markov chain based SBR models, e.g., FPMC, FOSSIL , conduct sequence modeling and prediction only by considering user’s last behavior. Lately, RNN-based models the treat historical behaviours of each user as a strictly-order, temporally dependent sequence like linguistic sentences. These methods like GRU4REC  surprisingly increase the performance in many real scenarios because of their effectiveness in storing short-term information. However, they assume that the adjacent items in a session have a fixed sequential dependence which is unable to capture the user interest variations, as a result, they are prone to introduce wrong dependencies. To address this issue, later models that fusing self-attention mechanisms like SASREC, were proposed. Guo et al.  further improve the attention-based approaches by introducing specialized human sentiment factors. The aforementioned attention-based methods prefer to model unidirectional message transformation between adjacent items in a sequence. Such transformation may lose the insight of the relevant information of the whole sequence. For example, in a music player APP, although a user may randomly play an album or a certain type of music, which will generate different playback records, it does not mean that the user’s interest has changed. In other words, strictly modeling the user’s local click records and ignoring the global relationship may lead to overfitting. To resolve the problem in attention-based methods, GNN-based models like SR-GNN  and GC-SAN  utilize graphs to capture the coherence of items within a session due to their powerful ability to represent structured data and adopt attention layers to learn long-term dependence. Despite the leading performance of GNN-based models compared with traditional SBR methods, great challenges remain.
Challenge1: The user’s interests are extensive and hierarchical, which can be expressed as a power-law distribution of items clicked by users. The existing session-based recommendation methods learn the representations in Euclidean space, which can’t effectively capture the information of such hierarchical, or in other words, tree-structured data.
Recent studies have proved that hierarchical data can be better explained under Non-Euclidean geometry of low-dimensional manifolds. But in the GNN-based methods, introducing Non-Euclidean transformation will result in the discrepancy between Euclidean and Non-Euclidean space when aggregating neighbouring messages and applying attention mechanism.
These challenges remain prevalent in real-world recommendation scenarios since it has been demonstrated that users’ behaviors like clicking or purchasing have prototypical characteristics of complex structures, which are generally power-law distributed [32, 19, 29]. Aforementioned, data with hierarchical structure could be well represented in hyperbolic space. This has motivated representation learning in hyperbolic space to effectively capture the information of the user behaviors with hierarchical property . Furthermore, the hyperbolic representations can naturally capture the similarity and hierarchy by their distance. To illustrate the difference between Euclidean and hyperbolic space, we visualized them in Figure 2 and Figure 2. In a two-dimensional Euclidean space as showed in Figure 2, the number of nodes increases polynomial in the center to the radius. By contrast, in two-dimensional hyperbolic space in Figure 2, the number of nodes increases exponentially in the center to the given radius, and the hyperbolic space has more powerful (exponential-level) representation ability than then Euclidean space. In conclusion, given the same radius, hyperbolic space has larger space, thereby including more nodes. Therefore, the general representational capacity of Euclidean space can be summarized as a square level, which can cause high distortion if we model the data of hierarchical relational users’ preferences.
To overcome these session-based challenges above, in this paper we propose a novel graph neural network framework, namely Hyperbolic Contrastive Graph Recommender (HCGR), upon hyperbolic space, specifically, Lorentz hyperbolic space for its simplicity and stability, to optimize the underlying hierarchical embeddings. First, we embed items with dense and effective representations, which are predisposed to preserving their internal hierarchical properties in Lorentz hyperbolic space. To ensure the correctness of the necessary representations’ transformation, we utilize some specific operations based on the Lorentz hyperboloid model. Second, we construct an improved graph neural network framework and a novel message propagation mechanism to model the preferences in user behavior sequences. To enable the model better distinguish users’ preferences for different items, we propose an adaptive hyperbolic attention calculation method. Third, we introduce contrastive learning to optimize the model by considering the distance between positive and negative samples in hyperbolic space.
Overall, we summarize our main contributions of this work as follows:
We exploit hyperbolic item representation for session-based recommendation, to the best of our knowledge, our method is the first one to extract hierarchical information of user behaviors within a session under hyperbolic space for SBR tasks.
We design a novel attention calculation approach in hyperbolic space to deal with graph information aggregating, which can’t be effectively implemented by the existing aggregation methods in Euclidean space.
We introduce contrastive learning to optimize the item representation by considering the geodesic distance between positive and negative samples in Lorentz hyperbolic space.
We conduct extensive experiments on three public datasets and one financial service industrial dataset, which show that our Lorentz hyperbolic session-based recommendation framework can achieve better performance compared to the state-of-the-art SBR methods in terms of , and , where .
The structure of this paper is as follows. Section 2 presents the related work, including session-based recommendation and hyperbolic learning. Section 3 introduces the preliminary work of this paper, including the definition of the problem and some basic knowledge of the Lorentz hyperbolic space. Section 4 introduces the implementation details of HCGR framework, and section 5 gives the experimental results. Finally, conclusions are set out in section 6.
2 Related work
In section 2.1
we review a line of representative works on session-based recommendation, including traditional MCs(markov chains) based models, RNNs(recurrent neural networks) based models, attention-based models, and GNNs(graph neural networks) based models. Then, in section2.2 we discuss related hyperbolic representation learning methods related to our proposed HCGR.
2.1 Session-based Recommendation
2.1.1 Markov chain models
Early sequential recommendation methods mainly rely on Markov chains. For example, FPMC  combines MF(matrix factorization) and MC to the learn general preference and local interest of users for the next basket recommendation. HRM  applies non-linear operations to extract more complex pattern of both user’s sequential behavior and interests. Fossil 
fuses similarity-based methods with Markov chains to conduct personalized sequential recommendations. A shortcoming of MC-based models is that it is difficult to learn long-term dependencies because MC-based models assume that the next state is only related to the previously nearest state. Although some high-order Markov models can associate the next state with several previous states, they consume high computational cost[23, 42].
2.1.2 Recurrent neural networks models
In recent years, researchers adopt RNNs to capture time dependency in temporal data. The first RNN-based sequential recommendation method is GRU4REC 
, which uses GRU(gated recurrent unit) to capture long-term dependencies among sessions. Leveraging a novel pair-wise ranking loss, GRU4Rec significantly outperforms MC-based approaches. Inspired by GRU4REC , MV-RNN  incorporates visual and textual information to alleviate the item cold start problem. Furthermore, ROM  utilizes an interactive self-attention mechanism to adaptively reorganize the entity memory and the topic memory for the rating prediction task. However, RNN-based methods assume that the adjacent items in a session have a fixed sequential dependence, which may generate wrong dependencies and introduce noises in real-world scenarios like music playing.
2.1.3 Attention mechanism
Recent models with attention mechanism  perform particularly well in sequential recommendation. Li et al. explores a hybrid encoder with attention mechanism to model users’ sequential behaviors and user’s interests in the same session . A short-term attention priority model STAMP  is proposed, which can capture both personal interest from the long-term memory of the session context, and user’s current interest from the short-term behaviors. SASREC  effectively captures users’ long-term preferences from the sparse and dense datasets, and FDSA  puts the features of behaviours and items into two distinct independent blocks of self-attention to model the transition patterns of the items and the transition patterns of the items and achieve remarkable effects.
2.1.4 Graph neural networks
Most advanced sequential recommendation models apply the self-attention mechanism to capture user behavior relations in a long sequence. However, it is a challenge to find out both implicit and explicit relation between adjacent behaviors. GNNs can find out such relations effectively [16, 42] and can capture complex interaction of user behaviors. For example, SR-GNN  instructively constructs a digraph for each sequence and CS-SAN  further incorporates self-attention mechanism to generate the representation of the constructed digraph. In addition, Wu et al. focuses on users in the session and models their historical sequence by dot-attention mechanism . FGNN  proposes a weighted attention layer and a readout function to learn item embeddings and session embeddings for next item recommendation. Recently, Ma et al. utilizes memory models to capture both the long term and short-term user behaviors . Wang et al. introduces a new GNN-based method that can learn global relations between items . Chen et al.  utilizes several ways to reduce the information within message propagation. GNN-based methods have yielded many fruitful results, but the existing methods generally model user behavior in tangent space, and the representation learned in tangent space is limited to capturing attributes of shallow properties and lack of hierarchy. In this work, we aim to learn hierarchical item representation in Lorentz hyperbolic space, and we want to find deep user behaviors patterns in session-based recommendation.
2.2 Hyperbolic Learning
Recently, many studies have shown that complex data may exhibit a highly non-Euclidean structure [2, 9]. Researchers are increasingly considering building complex neural networks on Riemannian space, in which the hyperbolic space with negative constant curvature is an attractive option [46, 26]. In many domains, such as sentences in natural language , social networks , biological protein graph , etc., data usually have a tree-like structure or can be represented hierarchically, and hyperbolic space is equipped to model hierarchical structures naturally and effectively . Due to its strong representation ability, hyperbolic space has been applied in many application areas [22, 4, 5, 7, 10]. For instance, Liu et al.  proposed Hyperbolic Graph Convolutional Networks used for graph representation learning by combining the expressiveness of hyperbolic space and Graph Convolutional Networks. Chen et al.  proposed a hyperbolic interaction model for multi-label classification tasks. These works have shown the advantages and effectiveness of hyperbolic space in learning hierarchical structures of complex relational data.
Noticing the potential of hyperbolic space in learning complex interactions between users and items, many researchers have tried to apply hyperbolic learning to recommendation systems [3, 34, 25, 20, 11]. Chamberlain et al.  proposed a large-scale recommender system based on the hyperbolic space, which can be scaled to millions of users.  considered constructing multiple hyperbolic manifolds to map the representation of user and ad, and proposed a framework that can effectively learn the hierarchical structure in users and ads based on the hyperbolic space. Ma et al.  proposed a recommendation model in the hyperbolic space for Top-K recommendation. Li et al.  presented the Hyperbolic Social Recommender which utilized hyperbolic geometry to boost the performance. Wang et al.  proposed a novel graph neural network framework (HyperSoRec) combing hyperbolic learning for the social recommendation.
In this section, we introduce basic knowledges about hyperbolic geometry and graph neural networks works related to our proposed HCGR.
3.1 Hyperbolic Geometry
Hyperbolic space is a Riemannian surface with negative curvature. Several hyperbolic geometric models have been widely used, including Poincare disk model [11, 36], Klein model  and Lorentz (hyperboloid) model . All these hyperbolic models are isometrically equivalent, i.e., any point in one of these models can be transformed to another point with distance-preserving transformations . In this paper, we choose the Lorentz model as the framework cornerstone, because of the numerical stability and calculation simplicity of its exponential/logarithm maps and its distance function. In hyperbolic geometry, we use the Lorentz formulation to model the network, which is found to be more stable for numeric optimization patterns . We want to learn -dimensional user and item embeddings.
A -dimensional hyperbolic space is a Riemannian manifold with a constant negative curvature, which is denoted by . The negative reciprocal of the curvature is denoted by where . The Lorentz representation is defined by the pair and
where is the Lorentz inner product given by
and the metric matrix is given by
The distance function induced by the metric is
For any pair of points , the tangent space at point is a -dimensional Euclidean space. The elements of
are referred to as tangent vectors and satisfying
3.2 Graph Neural Network
GNNs are neural networks that can handle graph-structured data directly. They are often applied in classification, link prediction and graph classification tasks. In this paper, we focus on graph classification, because we formulize each user’s behavior to a graph and we want to learn a representation from it rather than a single node.
Let denotes a given graph, where and are the set of the nodes and edges respectively, where represents the feature vector of , which is the initial embedding of node
. To specific, we formulate the graph classification task as follow. Our work is to learn a classifierand the graph-level representation to predict the label of the graph. Given a collection of graphs and the corresponding labels .
GNNs use the structure of graph and the original feature of each node to learn its corresponding representation. The learning process is to take a node as the center, and iteratively aggregate the neighborhood information along edges. The information aggregation and update process can be formulated as follows:
where represents the embedding of node after -th layer aggregator and is neighborhood of node . The information aggregation function aggregates the information from the neighborhood information and passes it to the target . The update function calculates the new node statues from the source embedding and the aggregated information .
After steps of information aggregation, the final embedding gather the -hop neighborhood and the structure information. For the graph classification task, readout function generates a graph level embedding by gathering the embeddings of all node in the final layers:
In this section, we describe the implementation details of the HCGR framework. First, in section 4.1 we illustrate the notations used in this paper and define the session-based recommendation task. In section 4.2, we transform the user behaviors within a session into a session graph, and present the HCGR’s overall pipeline in Figure 3. In section 4.3, we introduce the embeddings in Lorentz hyperbolic space. Next, we describe the novel attention mechanism that is especially designed for hyperbolic geometry in section 4.4. After learning the embeddings, we set up the hyperbolic attention mechanism to construct representation of user behaviors (section 4.5).Finally, we describe the contrastive learning with hyperbolic space distance (section 4.6).
4.1 Notation and Problem Definition
A session-based recommendation task is constructed on historical user behavior sessions, and makes predictions based on current user sessions. In this task, there is an item set , where is the number of items and all items are unique. Each session is composed of a series of user’s interactions, where represents an item clicked at the -th position in and represents session’s length. Our purpose of session-based recommendation is to predict the item that the user is most likely to click on next time in a given session .
For each given session in the training process, there is a label as the target. In the training process, for each item in given session, our model learns the corresponding embedding vector , where is the dimension of vector
. Our model outputs a probability distributionover the given session , where the item with Top- value will be regarded as the candidate for Top- recommendation.
|a hyperbolic space of dimension|
|a given session|
|the curvature of hyperbolic space|
|the negative reciprocal of the curvature|
|the tangent space at point with dimension|
|a item embedding in the Euclid space|
|the layer of graph neural network|
|a item embedding in the hyperbolic space|
4.2 Behaviors Graph
Because graph neural network can’t deal with session directly, the first thing we need to do is converting a given session to the session graph . According to the analysis of datasets, it is very common for users to click the same item multiple times within the session. Because the user’s behavior is chronological and the same item may be clicked multiple times, we choose the weighted directed graph to represent the changing process of the given session . All the sessions will be converted into session graphs. To show this process more clearly, we show the process of this session converter in Figure 4. We use to denote all weighted directed edges set. Its elements are composed of , where , is the item clicked at timestamp , respectively, denotes the weighted directed edge between and . Note that if the node does not have a self-loop, we will add a self-loop with weight to it. Each node represents the unique item in the session and the features are initialized in the Lorentz hyperbolic space which introduced in section 4.3.
4.3 Embeddings in Lorentz Hyperbolic Space
We use the representation from Lorentz hyperbolic space for item embedding. The is the reciprocal of curve , which treated here as a trainable parameter and initials empirically. Then we fix the origin
and use it as a reference point. The embeddings are initialized by sampling the Gaussian distribution on the Euclidean spaceof the reference point o.
We denote the mapping between hyperbolic spaces and tangent spaces as exponential map and logarithmic map, respectively. The exponential map is a map from subset of a tangent space to . The logarithmic map is the reverse map that maps back to the tangent space . For any and satisfying and , the exponential map and logarithmic map are defined as follows:
where is the Lorentzian norm.
4.4 Hyperbolic Graph Attention Network
Following the mapping layer, how to model session graph and mine user preferences is the key issue. Users typically click on several items they like, and these items have rich hierarchical structure. As a result, we propose a novel information aggregation with attention mechanism in hyperbolic space to capture the influence of different items on user preferences during the process of information propagation. Hyperbolic space can better represent item[3, 34], but we still face a technical challenge, the traditional hyperbolic model does not define the necessary vector operation process, such as vector addition and multiplication etc. Inspired by previous works [3, 34, 39], we formulate the multiplication and addition operation in hyperbolic space as follow:
where is the Parallel Transport: for two point on the Lorentz space , the parallel transport of a tangent vector to the tangent space is:
Non-linear activation with different curvatures is proposed as follow:
where , is the hyperbolic curvature at layer , respectively. To specific, we project the embedding from the tangent space to via exponential map according to the Eq(4.8). Then we perform the addition and multiplication operation according to the above equation.
The crucial idea of traditional GNNs is to learn representations in the given graph by iteratively aggregating and capture multi-hop neighborhood structures and features. The process of information aggregation usually consists of two parts: feature transformation and nonlinear activation. Recent studies have shown that, compared to the simple average aggregation method, the gain with features transformation and non-linear activation is rarely, and may even bring negative gain. In addition, these two operations may lead to significant over fitting of highly sparse user behavior. Based on these studies, we remove the unnecessary feature transformation and non-linear activation to accelerate the training, inference and reduce the complexity of our framework.
To make better use of the representation ability of Lorentz space, we redesigned a way of information aggregation in hyperbolic space. We refer to the ideas of GCN and GAT to calculate the attention weight between target node and neighbors respectively. The detailed calculation way is shown as follow:
where is matrices, and is a constant number. The learning process is to take a node as the center, and iteratively aggregate the neighbors information along edges. For each node , in hyperbolic mechanism, all attention coefficients of their neighbors can be calculated as Eq(4.14). In order to use these attention coefficients, a linear combination for the neighbors is used for updating the embeddings of the nodes.
To take full advantage of higher-order relationships, we need to stack multiple hyperbolic attention layers together.
4.5 Hyperbolic Attention Mechanism
After we obtain the graph level representation , we want to utilize self-attention mechanism to better capture user’s preference. Self-attention is an important part of attention mechanism, it has yielded many fruitful results such as:,,. The self-attention mechanism can calculate the global dependence between user behavior and capture the item transformation relations of the whole session sequences. The original self-attention mechanism does not define in hyperbolic space, so we extend the self-attention mechanism to hyperbolic space and we formalize the hyperbolic self-attention mechanism as follows:
where , and are matrices. It will receive the query (), key (), and value (), and calculate the similarity between each element in the session through the scaled dot-product, so as to characterize the user’s long-term preference, where d is the dimension of the input vector and is the scale factor, which is used to prevent the gradient vanishing problem caused by the large value after the dot product.Correspondingly, Element-wise Feed-Forward is also extended to hyperbolic space and it is given by:
where and are matrices, and are
-dimensional bias vectors. It takes full account of the interaction between the dimensions of various vectors through nonlinear activation function and linear transformation. A skip connection after the feed forward network, which makes the model reduce the loss of information and takes advantage of the low-layer information. For simplicity, we define the entire hyperbolic self-attention mechanism above as:
Recent studies have shown that, different layers of self-attention mechanism may capture different types of features, so it is necessary to increase the number of layers appropriately to enhance the model expression. The multi-layer hyperbolic self-attention mechanism is define as:
Finally, the hyperbolic self-attention mechanism output is . After -th adaptive hyperbolic self-attention blocks, we obtain the long-term attentive session representation . The short-term interest describes the current preferences of users. It is based on several items recently visited as the basis for prediction. The next behavior of users is often closely related to his recent interests. In order to better model the relationship in the whole session, we set up a gated mechanism to capture both long-term and short-term preference.
where denotes the embedding corresponding of the last item in the given session S.
Finally, after we get a unified preference representation , we make a recommendation score for each element in the item set .
where is the recommendation probability of our framework for item . For the session-based recommendation task, we select the highest probabilities item from item set as final result according to .
4.6 Contrastive Learning
By projecting the item embedding into hyperbolic space, we empower the performance of our framework. In the recommendation scenario, there are many similar items, but users usually only choose their favorite items, so if we can let the model distinguish this subtle distinction, it may significantly improve the recommendation ranking performance of our framework. Inspired by the successful practice of contrastive learning, we introduce contrastive learning in an innovative way into the framework in order to increase the modeling of user behavior. Compared with simple contrastive learning, the operation of our framework is calculating in hyperbolic space, which will be somewhat more complicated. Specifically, we want to make the best use of the distance between items in hyperbolic space through contrastive learning, then the recommendation model perceives more subtle distinction and improve the ranking performance.
We formulate our objective into two parts, the first part
is cross-entropy loss function, which has been widely used in recommender system. The second part is the contrastive ranking losswith margin. The purpose of is to separate the positive and negative pairs up to a given margin. When the margin is reached, the pairs of items are considered to be properly segregated and with little loss. This enables the model keep focus on the pairs of items that are not near the margin and the margin separation is optimized in Lorentz hyperbolic space.
where and control the magnitude of the cross entropy loss and contrastive ranking loss respectively.
In this section, detailed experiments will be conducted to assess the performance of the HCGR framework. We intend to answer following questions:
RQ1: How does our proposed method perform comparing with the state-of-the-art methods?
RQ2: How the different components (i.e., Lorentz transportation, multi-hop graph aggregation and contrastive learning) affect the performance of HCGR?
RQ3: Can HCGR present reasonable explanation with regard to predicting user preference and get better recommendation results ?
In particular, we first describe the datasets and experimental configuration (section 5.1). Then we compare the effectiveness of HCGR with several comparison methods (section 5.2). In section 5.3, we analyze in detail the generalization capability and the possibility of migration of the HCGR. Lastly, section 5.4, we set the case study and visualize the embedding in hyperbolic space (section 5.5).
5.1 Experimental Setup
5.1.1 Datasets and Metrics
We evaluate different recommenders based on four publicly available datasets, three of which are public benchmark datasets, i.e., , , and -. Specifically, 111https://recsys.acm.org/recsys15/challenge/ is the competition dataset of Recsys challenge 2015. It contains e-commerce website click within six months and related information. 222http://millionsongdataset.com/lastfm/ dataset contains a set of users from online music service, which describes tagging and the music listening details of user. The -333http://recsyswiki.com/wiki/Groceryshoppingdatasets dataset is a grocery dataset published by ACM RecSys, it covers goods ranging from food, office supplies to furniture. The fourth dataset is the financial service scenarios dataset , which is an industrial online recommendation platform in the Ant Group. dataset describes users’ interests and preferences in financial products such as debit, trust, accounting, which contains more than 5.6 million interactions from 691,701 users and brings more challenges compared with the three public datasets. The data statistical status after preprocessing is summarized in Table 2, where Avg.I/user and Avg.I/item denote ”average interaction per user” and ”average interaction per item”, respectively. To filter noisy data, we filter out items that appear less than 3 times, and then remove all user’s behaviors less than 3 items on four datasets. After preprocessing, we split user behaviors into three parts, i.e., we randomly pick 80 as training set, 10
as validation set for hyper-parameter tuning, and the remaining part for evaluating the performance of the model. Furthermore, to prevent overfitting, we set the patience argument to be 10 in the early stopping mechanism which denotes how many epochs we want to wait after the last time the validation metrics improved before breaking the training loop.
To fairly compare the generalization performance of each model, we evaluate for each user on his/her performance in the test set by adopting three recognized metrics: , and . Here, we choose to show the different metrics for , and .
If one or more element of the label is shown in the prediction results , we call it a hit. The is calculated as follow:
where , denotes the indicator function and is an empty set. A larger value of reflects the accuracy of the recommendation results.
Normalized Discounted Cumulative Gain is a ranking based metric, which focuses on the order of retrieval results and is calculated in the following way:
where is a constant to denote the maximum value of given and denotes an indicator function.A large value reflects a higher the ranking position of the expected item.
Mean Reciprocal Rank(MRR) when the r item is not in the higher position, the reciprocal is set to 0. It is formally given by:
where denotes the position of the item in . is a normalize ranking take into account the order of recommendation list . A large value reflects a higher ranking position of the expected item.
5.1.2 Comparison Methods
To demonstrate the performance of HCGR, we consider the following representative methods for performance comparisons:
FPMC  - a classical markov-based model, which considers the latest interaction.
FOSSIL - a classical markov-based model, which captures personalized dynamics.
GRU4Rec  - a representative RNN-based method for session-based recommendation, it stacks multiple GRU layers for session-parallel mini-batch training.
NARM  - a hybrid encoder with attention mechanism to model sequential behaviors in the current session.
SASRec  - a self attention-based sequential recommender, which utilizes relatively few actions and considers long-range dependencies.
STAMP  - a short term behavior priority attention-based method.
SRGNN  - a graph-based recommender modeling session to learn item representations.
GC-SAN  - an improved version of SRGNN, which uses a GNN and multi-layered self-attention mechanism to compute the sequence level embeddings.
FGNN  - a graph-based method, which uses a weight attention network to compute the graph level embeddings.
LESSR  - a session-based recommender with GNN, which innovatively utilizes auxiliary graph to generate item representation.
HCGR - our approach with novel attentive information aggregation, which utilize contrastive loss to optimize the model by considering the distance between positive and negative samples in hyperbolic space.
In this experiment, we set the maximum length of session to be , and the embedding dimension to be =128 for all datasets, the initial learning rate is uniformly set to 0.001 ,the linear schedule decay rate of 0.5 of every 3 epochs and penalty is
. All parameters are initialized by Gaussian distribution with mean value of 0 and standard deviation of 0.1. The model cooperates with the Adam optimizer to complete the training.
5.1.3 Data exploration
As discussed in section 2.2, complex data with tree-like structure, i.e., the data obeys a power-law distribution, is effectively explained in the hyperbolic space. Therefore, we check the data distribution used in our experiment to verify the appropriateness of dataset selection. We present the distribution of number of interactions between users and items as illustrated in Figure 5. Power-law distribution is observed in three public datasets and the dataset. In the user-item interactions, a majority of users interact with items very few times, meanwhile, most items are with few clicks. Such results demonstrate the tree-like structure of our dataset, which is supposed to have better representations in the hyperbolic space for session-based recommendation.
5.2 Overall Performance (RQ1)
|* Realtive improvemens are calculated by comparing with the second best performance|
The experimental results of all comparison methods in session-based recommendation are presented in Table 3. The best results of each column are highlighted in boldface. As can be observed, HCGR outperforms the best baselines with more than 4.5% performance improvement on average on all datasets. From the results in Table 3, we can draw the following main findings:
The RNN-based approaches which capture sequential dependency in a session(i.e., GRU4REC, NARM) remarkably outperform the traditional models that rely on Markov chains(i.e., FPMC, FOSSIL). This phenomenon proves that capturing sequential effects is a key factor for session-based recommendation as user’s session-based behaviors are usually included in a short period and are likely to be temporally dependent.
The attention-based models(i.e., NARM, SASRec, and STAMP) that involve attention mechanism get higher performance compared with that do not(i.e., GRU4REC) in all evaluation metrics. This is because NARM, STAMP, and SASRec can extract the shift of user interest within sessions and get the main purpose in the current session by incorporating an attention mechanism, which captures personal interest from the long-term memory or just models the user’s current interest from the short-term behaviors. This phenomenon indicates that RNN-based approaches with the assumption that adjacent items in a session have a fixed sequential dependence may generate wrong dependencies, which further results in recommendation bias. This could be alleviated by involving the attention mechanism.
The GNN-based models (i.e., GC-SAN, FGNN) achieve better performance than RNNs-based models with or without attention mechanism due to the remarkable capacity of graph neural networks to capture complex interaction of user behaviors and describe the coherence of items in a session, which are ignored by RNNs-based models and such ignorance leads to overfitting in RNNs-based models.
Our proposed HCGR consistently outperforms all the comparison models on all datasets. Compared with FGNN and LESSR, our model involves an advanced hyperbolic learning component to more effectively capture the coherence and hierarchy representations of the user behaviors within the Lorentz hyperbolic space, which ensures the correctness of the necessary representations’ transformation. Furthermore, we use a novel graph message propagation mechanism with adaptive hyperbolic attention calculation to model user’s preferences in session behavior sequences. In addition, we introduce contrastive learning to optimize the model by considering the distance between positive and negative samples in hyperbolic space, which can help learn better item representations.
5.3 Ablation Study (RQ2)
5.3.1 Effect of Lorentz Transformation
To demonstrate the effectiveness of the proposed hyperbolic learning framework for session-based recommendation, we conduct the ablation experiments by combining the Lorentz transformation with several baseline Euclidean SBR models, including FPMC, GRU4Rec, SASRec, and SRGNN. Besides, we also compare the performance of HCGR with ECGR (Euclidean Contrastive Graph Representation) by removing the Lorentz transformation from the hyperbolic contrastive graph representation learning framework shown in Figure 3. The experimental results are shown in Table 4. The postfix means the corresponding model is combined with hyperbolic learning to extract the hierarchy information contained within the SBR datasets. From Table 4, we can draw the following conclusions:
The performance of all models improves significantly on the four datasets when combining the Lorentz transformation with the Euclidean SBR models, which demonstrates that the hierarchy information from the power-law like session-based recommendation data is essential for predicting the user behavior, while such information is just ignored by the traditional SBR models built upon Euclidean space. Furthermore, the improvement of Markov-based method(i.e., FPMC) and attention-based method(i.e., SASRec) is more obvious than that of RNN-based(i.e., GRU4Rec) and GNN-based(i.e., SRGNN) method.
Our proposed hyperbolic contrastive graph representation learning method HCGR achieves the best results over all comparison models with or without Lorentz transformation, but the performance of ECGR drops evidently when replacing the Lorentz transformation with Euclidean transformation on all datasets. Besides, we find that the ECGR outperforms most baseline SBR models coupling with Lorentz transformation, which indicates the advantage of the proposed contrastive graph representation learning method.
5.3.2 Effect of Graph Aggregation Method
To further investigate the advantage of the proposed adaptive hyperbolic graph aggregation method that utilizes multi-hop adjacent information, we conduct an ablation study by comparing different graph aggregation information approach within the framework of hyperbolic contrastive representation learning on four datasets. HCGR_GCN refers to a model that the traditional spectrum-based graph convolution method is used to transport messages among adjacent neighbors, while HCGR_GAT refers to a model that the graph attention-based method is used to aggregate adjacent information. The experimental results are shown in Figure 6. we can draw the following conclusions:
The inductive attention-based graph convolution model(i.e., HCGR_GAT) remarkably outperforms the transductive spectrum-based graph model(i.e., HCGR_GCN) on all datasets, which indicates that treating neighbours deferentially and flexibly is essential to filter noisy information during message aggregation.
No surprisingly, HCGR with our proposed multi-hop adjacent information aggregation method achieves the best performance. Comparing to GAT, the main improvement of our proposed aggregation method relying on multi-hop aggregated message during graph node representation optimization is fully used, which indicates that low-order and high-order mutual graph information are both critical for final prediction. Such low-order mutual information is just ignored by GAT-like models.
5.3.3 Effects of Contrastive Ranking Loss
Diversity has become an important evaluation indices in recommendation scenario. In order to investigate the effectiveness of contrastive learning on the performance of our proposed hyperbolic graph representation learning framework, we conduct an experimental analysis by removing the contrastive ranking loss. To specific, HCGR_CE means that contrastive ranking loss is removed from Eq (4.25) while keeping other settings same as HCGR. The experimental resutls are shown in Table 5 and Table 6, we can draw the following observations:
In all datasets, the contrastive ranking loss optimization model HCGR outperforms the cross-entropy loss optimization model(HCGR_CE) as regard to ranking evaluation metrics(, ), which indicates that the contrastive ranking loss can distinguish the subtle distinction between items within sessions and improve recommendation diversity.
With regard to the accuracy of recommendation results, there is no obvious difference between the performance of the two models, which indicates that our contrastive ranking loss can improve the ranking performances without losing recommendation accuracy.
5.3.4 Effects Of Embedding Size
We explore the impact of embedding size on several evaluation indices as such size significantly affect the representation ability. We conduct the experiment on and datasets as their generalization and representation. The results are plotted in Figure 7-8. We have the following observations:
HCGR outperforms all comparison SBR methods with most embedding sizes in all indices. Especially for small embedding size, such as 32, our model still can achieve better and robust results on these indices, which indicates introducing hyperbolic transformation can capture latent hierarchy property and boosts model performance.
It is also observed that a proper embedding size is essential for graph node representation. When the embedding size is too small, it can’t fully express node information and result in poor performance. On the opposite, a large embedding size may induce overfitting on the dataset.
5.4 Case Study (RQ3)
5.4.1 Representation Analysis
We set up the case study to explore whether our model can learn the hierarchical structure of user behavior. Whether the hierarchical structure in the data can be fully learned will affect the performance of the model, and this kind of hierarchical structure can be reflected by calculating the distance between the representation and the origin. HCGR, ECGR are calculated in two different geometries, we use gyrovector space distance and tangent distance respectively to calculate the distance from the target point to the origin. We set up three boundaries in Euclidean space and Lorentz space, and divide the representation into four regions according to their distance from the origin. For example, the item of region 1 is the closest to the origin, whereas the item of region 4 is the farthest from the origin. To intuitively reflect the different popularity of items in different regions, we count the interaction times of nodes in all regions of the four datasets. We visualize the statistics as Figure 9. From the results, it can be seen that the average number of interactions of items from region 1 to region 4 has decreased, which shows that both of our approaches ECGR and HCGR can model the hierarchical structure of session behavior. In addition, in all datasets, the average number of interactions of items with HCGR in region 1 is higher than that with ECGR, while the average number of interactions of items with ECGR in region 3 and 4 is higher than that with HCGR. Compared with ECGR, HCGR can better distinguish the items with different popularity and learn the hierarchical structure, which indicates that hyperbolic space is more suitable for embedding hierarchical data than Euclidean space for session-based recommendation tasks.
5.4.2 Attention Analysis
Taking advantage of the attention mechanism, we visualize the attention weight between user behaviors as shown in Figure 10, which reflects the different influences within the same session on the two models (HCGR, ECGR). We randomly select three different sessions of length 10(), 30() and 50() respectively from dataset(test set). For a same session in the heatmap, the above one is the attention weight between the related items and the next item user most likely to click modelled by ECGR, while the following one is the corresponding attention weight modelled by HCGR. From the heatmap, we discover that not all the behaviours in the same session equally contributing during the generation of the recommendation. In addition, we also find that the attention weight of HCGR for session behavior is higher than that of ECGR in many key positions. Specifically, HCGR will give higher scores with the increase of the scores given by ECGR, and HCGR can better distinguishes the item importance. It proves that hyperbolic space can better represent the hierarchical structure of data, thus making the attention mechanism capable of adaptively measuring the influence of session behavior.
5.5 Embedding Analysis and Visualization
We visualize the item embeddings in 2-dimension and 3-dimension on , , - and datasets repectively according to Figure 11. (a) - (d). The popularity of the items is represented according to the different colors, decreasing with the color from red to green. Before training, we randomly initialize all the item embeddings. As shown in Figure 11, it is obvious that item embeddings present a hierarchical structure based on item popularity after training. On the dataset, we can observe such a clear hierarchical representation, with the most popular items in the center and unpopular ones stay away from the center of projection space. Similar results can also be obtained on other datasets.
The GNN-based model can not capture the hierarchical information effectively, which regularly appeared in recommendation scenarios. Enlightened by the powerful representation of non-Euclidean geometry which is proved to be able to reduce the distortion of embedding data onto power-law distribution, we proposed a hyperbolic contrastive graph recommender (HCGR), utilizing Lorentz hyperbolic space for item embeddings preserving their coherent and hierarchical properties. We design a novel hyperbolic graph message propagation mechanism due to the discrepancy between Euclidean and hyperbolic space during information passing. In addition, we introduce contrastive learning to enhance model performance by optimizing the distance between positive and negative samples of hyperbolic space, considering that distance in hyperbolic space can’t be expressed well by traditional loss, such as CE, BPR loss. For future work, we will extend our method to the sequential recommendation which involves user profile and more item features. Besides, we will learn a parsimonious representation of symbolic data by embedding the dataset into spherical or product space and optimize the process of matrix multiplication in non-Euclidean geometry.
-  (2020) Holography on tessellations of hyperbolic space. Physical Review D 102. Cited by: §1.
Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine 34 (4), pp. 18–42. External Links: Cited by: §2.2.
-  (2019) Scalable hyperbolic recommender systems. arXiv preprint arXiv: 1902.08648. Cited by: §2.2, §4.4.
Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4869–4880. Cited by: §2.2.
Hyperbolic interaction model for hierarchical multi-label classification.
The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7496–7503. Cited by: §2.2.
-  (2020) Handling information loss of graph neural networks for session-based recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1172–1180. Cited by: §2.1.4, 10th item.
Self-supervised hyperboloid representations from logical queries over knowledge graphs. In WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pp. 1373–1384. Cited by: §2.2.
-  (2018) MV-rnn: a multi-view recurrent neural network for sequential recommendation. IEEE Transactions on Knowledge and Data Engineering 32 (2), pp. 317–331. Cited by: §2.1.2.
Non-euclidean dissimilarities: causes and informativeness.
Structural, Syntactic, and Statistical Pattern Recognition, Berlin, Heidelberg, pp. 324–333. Cited by: §2.2.
-  (2020) HME: A hyperbolic metric embedding approach for next-poi recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 1429–1438. External Links: Cited by: §2.2.
-  (2018) Hyperbolic neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 5350–5360. Cited by: §2.2, §3.1.
-  (2019) Hyperbolic attention networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Cited by: §3.1.
-  (2016) Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 191–200. Cited by: §1, §2.1.1, 2nd item.
-  (2020) Lightgcn: simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639–648. Cited by: §4.4.
-  (2015) Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939. Cited by: §1, §2.1.2, 3rd item.
-  (2018) Improving sequential recommendation with knowledge-enhanced memory networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 505–514. Cited by: §2.1.4.
-  (2018) Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 197–206. Cited by: §2.1.3, §4.5, 5th item.
-  (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. Cited by: §4.4.
-  (2010) Hyperbolic geometry of complex networks. Physical Review E 82 (3), pp. 036106. Cited by: §1.
-  (2021) HSR: hyperbolic social recommender. arXiv preprint arXiv: 2102.09389 abs/2102.09389. Cited by: §2.2.
-  (2017) Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1419–1428. Cited by: §2.1.3, 4th item.
-  (2019) Hyperbolic graph neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8228–8239. Cited by: §2.2.
-  (2018) STAMP: short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1831–1839. Cited by: §2.1.1, §2.1.3, 6th item.
-  (2020) Memory augmented graph neural networks for sequential recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 5045–5052. Cited by: §2.1.4.
-  (2021) Knowledge-enhanced top-k recommendation in poincaré ball. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 4285–4293. Cited by: §2.2.
-  (2006) Rebuilding community ecology from functional traits. Trends in Ecology and Evolution 21 (4), pp. 178–185. External Links: Cited by: §2.2.
-  (2017) Poincaré embeddings for learning hierarchical representations. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6338–6347. Cited by: §2.2.
Learning continuous hierarchies in the lorentz model of hyperbolic geometry.
Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, Vol. 80, pp. 3776–3785. Cited by: §3.1.
-  (2012) Popularity versus similarity in growing networks. Nature 489 (7417), pp. 537–540. Cited by: §1.
-  (2019) Rethinking the item order in session-based recommendation with graph neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 579–588. Cited by: §2.1.4, 9th item.
-  (1995) Introduction to hyperbolic geometry. American Mathematical Monthly 103 (2), pp. 185. Cited by: §3.1.
-  (2003) Hierarchical organization in complex networks. Physical review E 67 (2), pp. 026112. Cited by: §1.
-  (2010) Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web, pp. 811–820. Cited by: §1, §2.1.1, 1st item.
-  (2020) Multi-manifold learning for large-scale targeted advertising system. arXiv preprint arXiv: 2007.02334. Cited by: §2.2, §4.4.
-  (2021) HGCF: hyperbolic graph convolution networks for collaborative filtering. In Proceedings of the Web Conference 2021, pp. 593–601. Cited by: §1.
-  (2019) Poincare glove: hyperbolic word embeddings. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Cited by: §2.2, §3.1.
-  (2017) Attention is all you need. arXiv preprint arXiv:1706.03762. Cited by: §2.1.3, §4.5.
-  (2017) Graph attention networks. arXiv preprint arXiv:1710.10903. Cited by: §4.4.
-  (2021) HyperSoRec: exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation. Cited by: §2.2, §4.4.
-  (2015) Learning hierarchical representation model for nextbasket recommendation. In Proceedings of the 38th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 403–412. Cited by: §2.1.1.
-  (2020) Beyond clicks: modeling multi-relational item graph for session-based target behavior prediction. In Proceedings of The Web Conference 2020, pp. 3056–3062. Cited by: §2.1.4.
-  (2019) A neural influence diffusion model for social recommendation. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp. 235–244. Cited by: §2.1.1, §2.1.4.
-  (2019) Session-based recommendation with graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, pp. 346–353. Cited by: §1, §2.1.4, 7th item.
-  (2019) Personalizing graph neural networks with attention mechanism for session-based recommendation. arXiv preprint arXiv:1910.08887. Cited by: §2.1.4.
-  (2019) Graph contextualized self-attention network for session-based recommendation.. In IJCAI, Vol. 19, pp. 3940–3946. Cited by: §1, §2.1.4, 8th item.
-  (2017) Social collaborative filtering by trust. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (8), pp. 1633–1647. External Links: Cited by: §2.2.
-  (2019) Feature-level deeper self-attention network for sequential recommendation.. In IJCAI, pp. 4320–4326. Cited by: §2.1.3, §4.5.
-  (2020) Sentiment-guided sequential recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1957–1960. Cited by: §1.
-  (2020) Memory reorganization: a symmetric memory network for reorganizing neighbors and topics to complete rating prediction. IEEE Access 8, pp. 81876–81886. Cited by: §2.1.2.