Source code for the paper Residual Enhanced Multi-Hypergraph Neural Network.
Hypergraphs are a generalized data structure of graphs to model higher-order correlations among entities, which have been successfully adopted into various research domains. Meanwhile, HyperGraph Neural Network (HGNN) is currently the de-facto method for hypergraph representation learning. However, HGNN aims at single hypergraph learning and uses a pre-concatenation approach when confronting multi-modal datasets, which leads to sub-optimal exploitation of the inter-correlations of multi-modal hypergraphs. HGNN also suffers the over-smoothing issue, that is, its performance drops significantly when layers are stacked up. To resolve these issues, we propose the Residual enhanced Multi-Hypergraph Neural Network, which can not only fuse multi-modal information from each hypergraph effectively, but also circumvent the over-smoothing issue associated with HGNN. We conduct experiments on two 3D benchmarks, the NTU and the ModelNet40 datasets, and compare against multiple state-of-the-art methods. Experimental results demonstrate that both the residual hypergraph convolutions and the multi-fusion architecture can improve the performance of the base model and the combined model achieves a new state-of-the-art. Code is available at <https://github.com/OneForward/ResMHGNN>.READ FULL TEXT VIEW PDF
In this paper, we present a hypergraph neural networks (HGNN) framework ...
Hypergraph, an expressive structure with flexibility to model the
Graph representation learning has made major strides over the past decad...
This paper presents a novel version of the hypergraph neural network met...
Previous hypergraph expansions are solely carried out on either vertex l...
Depth completion aims to recover a dense depth map from the sparse depth...
With the rising number of machine learning competitions, the world has w...
Source code for the paper Residual Enhanced Multi-Hypergraph Neural Network.
The hypergraph structure consists of a set of vertices and hyperedges, in which a hyperedge can contain any number of vertices. In recent years, hypergraphs have attracted increasing attention in various computer vision tasks including 3D object classification
, 3D pose estimation, person re-identification , cross-modal retrieval , hypergraph-based image processing  and video segmentation .
In the meantime, multi-modal datasets are becoming more and more common since the increasing availability of multiple datasets acquired from different sources while describing the same category. When dataset for each modal can be represented by a hypergraph, this leads to the multi-hypergraph learning problem. Traditional multi-hypergraph learning models, such as tMHL  and iMHL [7, 18], utilize the hypergraph Laplacian to design the optimization schemes, which are shallow and require high computational cost.
HyperGraph Neural Network (HGNN)  is currently the de-facto method for hypergraph representation learning, after which the expressive embeddings are employed into diverse downstream tasks. Despite the success of HGNN, it fails to handle multi-modal dataset directly. HGNN aims at single hypergraph learning and uses a pre-concatenation approach when confronting multi-modal datasets, which leads to information loss since the inter-correlations between multi-modal hypergraphs are ignored.
HGNN also suffers the over-smoothing issue, a phenomenon that the model performance drops significantly as the number of layers increase . This degradation of learning limits HGNN to be a 2-layer model, of which the maximal exploitation of hypergraph structures can not be obtained.
In this paper, we propose the Residual enhanced Multi-HyperGraph Neural Network (ResMultiHGNN) to address the above issues. We summarize our contributions as follows: (a) We present the first Multi-Hypergraph Neural Network for direct and parallel learning in multi-modal datasets. (b) We enhance the traditional hypergraph convolutions with residual connection and input mapping, which eventually help to circumvent the over-smoothing issue in deep hypergraph models. (c) With the combined techniques, we present the first deep multi-hypergraph neural network for deep multi-modal learning. In the view-based 3D object classification task, our model achieves a new state-of-the-art.
Let the triplet denote a hypergraph, where is a set of vertices, each hyperedge is a non-empty subset of and is the features of vertices. Each row of denotes a -dimensional feature of a vertex. The degree of a hyperedge is while the degree of a vertex is defined as the number of hyperedges containing . Let and denote the diagonal matrices of the vertex degrees and hyperedge degrees respectively. The hypergraph can also be characterized by , where is the incidence matrix with each nonzero entry denoting .
HGNN utilizes the hypergraph Laplacian to design hypergraph convolution, where is defined as
Let and denote the learnable parameters and the input features in the -th layer respectively, then the message passing process of HGNN is formulated as
is the activation function.
Given a multi-modal dataset containing hypergraphs, HGNN pre-concatenates these hypergraphs into a larger hypergraph and then conducts the single hypergraph learning.
To better exploit the inter-relations between different modals, we propose the Multi-Hypergraph Neural Network (MultiHGNN), which include distinct branches to extract high-level information from each modal in parallel. As illustrated by Fig. 1, embeddings from multi-modal hypergraphs are combined together and then output for the downstream tasks. Notice that the gradients in each branch are back-propagated respectively, therefore the inner structure for each modal can be fully explored. In practice we use the simple Mean function to combine the multiple embeddings.
Formally, the output of MultiHGNN is formulated as
where is the hypergraph Laplacian of .
In computer vision, it is well known that residual connections  are the key components for making powerful and extremely deep networks. Drawing inspiration from GCNII , we enhance the vanilla hypergraph convolution with initial residual and identity mapping. Given the single hypergraph Laplacian , we define the propagation process of residual enhanced hypergraph convolution in the -th layer as
are hyperparameters and
is the identity matrix.
We present the ResHGNN by naively stacking multiple blocks of residual hypergraph convolution, which empirically circumvent the over-smoothing issue associated with HGNN.
We combine both techniques to build very deep multi-hypergraph neural network, denoted as ResMultiHGNN. In each branch, additional Linear transforms are added in the first and last layer, then the residual hypergraph convolutions are employed to propagate information over the distinct hypergraph structure. The deep embeddings from each branch are finally combined for downstream tasks.
We evaluate the performance of the proposed method on view-based 3D object classification task. We use the Princeton ModelNet dataset  and National Taiwan University (NTU) 3D model dataset as testing benchmarks. The ModelNet40 dataset is composed of 12,311 3D CAD models from 40 popular object categories, which is the publicly used subset of Princeton ModelNet dataset . We use the same training/testing splits as , where the training split contains 9,843 objects and the testing split contains 2,468 objects. The NTU dataset  includes 2012 3D objects from 67 categories, such as boat, motorcycle, train, and truck. We closely follow  and use the same training/testing splits. Details of the datasets are listed in Table 1.
For fair comparison, we employ the same shape features as , which are extracted from two multi-view based 3D shape descriptors, i.e. Multi-View Convolutional Neural Net Network features (MVCNN) 
and Group-View Convolutional Neural Network (GVCNN) features. The hypergraph of each modal is constructed based on respective features. Each hyperedge contains the 10 nearest neighbors of the central vertex/object. The multi-hypergraphs can be directly fed into the MultiHGNNs, whereas HGNNs only accept the pre-concatenated single hypergraph as input.
Note that the original training splits in both datasets contain unbalanced labels. We further conduct experiments by using only the balanced subset of original training labels and predict the results on the rest
samples. We implement all models with Pytorch. We release our code publicly in GitHub for reproducible experiments.
Table 2 reports the performance of proposed methods against HGNN with different depths, which can also be viewed as the ablation study of multi-fusion structure and residual hypergraph convolution. We visualize the results in Fig 2 for better comparison. Based on the table and the figure, we summarize the observations as follows: (1) Regardless of adding the residual hypergraph convolution or not, MultiHGNNs are consistently better than HGNNs, which verifies the effectiveness of the proposed multi-fusion architecture. (2) Residual connection can consistently enhance the performance of HGNN and MultiHGNN. It is worthwhile to note that with the number of layers increase, residual enhanced models maintain stable performance whereas performances of vanilla models deteriorate significantly. (3) ResMultiHGNN outperforms all other methods on both datasets.
We also point out that only using the balanced subset of training labels and testing on the rest all samples produce much better results than using full training labels. This indicates the importance of balanced distribution of labels in training samples for hypergraph learning.
Table 3 summarizes the classification accuracy of proposed ResMultiHGNN model against multiple recent state-of-the-art methods on ModelNet40 dataset. We see that ResMultiHGNN achieves a new state-of-the-art against the object recognition methods and the hypergraph learning methods, which demonstrates the power of multi-modal fusions and residual enhanced deep structures.
We further investigate the stability of our method against HGNN by modifying the ratios of training labels. We conduct all experiments with 8 different seeds and report the best performed models with optimal layers, as visualized in Fig. 3
. We observe that ResMultiHGNN consistently shows better performances than HGNN in all training ratios with gains around 2% and 3%. HGNN exhibits higher variance whereas ResMultiHGNN is slightly more stable, especially when training ratios are small.
In this paper, we propose the first Multi-Hypergraph Neural Network for multi-modal learning, and enhance it with the residual connections to build deep structures. Extensive experiments demonstrate the effectiveness of proposed Multi-fusion architecture and Residual Hypergraph Convolution. ResMultiHGNN enjoys both gains and shows stable and better results against HGNN.
On visual similarity based 3d model retrieval. Comput. Graph. Forum 22 (3), pp. 223–232. External Links: Cited by: §4.1.
International Conference on Machine Learning. External Links: Cited by: §1, §3.2.
The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, pp. 3558–3565. External Links: Cited by: §4.1, §4.1, Table 3.
2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 264–272. External Links: Cited by: §1, §4.1.
Semi-dynamic hypergraph neural network for 3d pose estimation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, C. Bessiere (Ed.), Note: Main track External Links: Cited by: §1.
PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (Eds.), External Links: Cited by: §4.1.