On the Use of Unrealistic Predictions in Hundreds of Papers Evaluating Graph Representations

12/08/2021
by   Li-Chung Lin, et al.
9

Prediction using the ground truth sounds like an oxymoron in machine learning. However, such an unrealistic setting was used in hundreds, if not thousands of papers in the area of finding graph representations. To evaluate the multi-label problem of node classification by using the obtained representations, many works assume in the prediction stage that the number of labels of each test instance is known. In practice such ground truth information is rarely available, but we point out that such an inappropriate setting is now ubiquitous in this research area. We detailedly investigate why the situation occurs. Our analysis indicates that with unrealistic information, the performance is likely over-estimated. To see why suitable predictions were not used, we identify difficulties in applying some multi-label techniques. For the use in future studies, we propose simple and effective settings without using practically unknown information. Finally, we take this chance to conduct a fair and serious comparison of major graph-representation learning methods on multi-label node classification.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/20/2020

Recovering Accurate Labeling Information from Partially Valid Data for Effective Multi-Label Learning

Partial Multi-label Learning (PML) aims to induce the multi-label predic...
research
11/29/2022

A Cross-Conformal Predictor for Multi-label Classification

Unlike the typical classification setting where each instance is associa...
research
11/11/2020

Multi-Label Classification Using Link Prediction

Solving classification with graph methods has gained huge popularity in ...
research
01/17/2022

Multi-winner Approval Voting Goes Epistemic

Epistemic voting interprets votes as noisy signals about a ground truth....
research
05/08/2020

Multi-Instance Multi-Label Learning for Gene Mutation Prediction in Hepatocellular Carcinoma

Gene mutation prediction in hepatocellular carcinoma (HCC) is of great d...
research
12/19/2019

A multi-label classification method using a hierarchical and transparent representation for paper-reviewer recommendation

Paper-reviewer recommendation task is of significant academic importance...
research
07/20/2022

MLMSA: Multi-Label Multi-Side-Channel-Information enabled Deep Learning Attacks on APUF Variants

To improve the modeling resilience of silicon strong physical unclonable...

Please sign up or login with your details

Forgot password? Click here to reset