Investigating the Robustness of Natural Language Generation from Logical Forms via Counterfactual Samples

by   Chengyuan Liu, et al.

The aim of Logic2Text is to generate controllable and faithful texts conditioned on tables and logical forms, which not only requires a deep understanding of the tables and logical forms, but also warrants symbolic reasoning over the tables. State-of-the-art methods based on pre-trained models have achieved remarkable performance on the standard test dataset. However, we question whether these methods really learn how to perform logical reasoning, rather than just relying on the spurious correlations between the headers of the tables and operators of the logical form. To verify this hypothesis, we manually construct a set of counterfactual samples, which modify the original logical forms to generate counterfactual logical forms with rarely co-occurred table headers and logical operators. SOTA methods give much worse results on these counterfactual samples compared with the results on the original test dataset, which verifies our hypothesis. To deal with this problem, we firstly analyze this bias from a causal perspective, based on which we propose two approaches to reduce the model's reliance on the shortcut. The first one incorporates the hierarchical structure of the logical forms into the model. The second one exploits automatically generated counterfactual data for training. Automatic and manual experimental results on the original test dataset and the counterfactual dataset show that our method is effective to alleviate the spurious correlation. Our work points out the weakness of previous methods and takes a further step toward developing Logic2Text models with real logical reasoning ability.


PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation

Logical table-to-text generation is a task that involves generating logi...

Unveiling the link between logical fallacies and web persuasion

In the last decade Human-Computer Interaction (HCI) has started to focus...

Empowering Language Understanding with Counterfactual Reasoning

Present language understanding methods have demonstrated extraordinary a...

Logical Inference for Counting on Semi-structured Tables

Recently, the Natural Language Inference (NLI) task has been studied for...

Neural Multi-Step Reasoning for Question Answering on Semi-Structured Tables

Advances in natural language processing tasks have gained momentum in re...

Logic2Text: High-Fidelity Natural Language Generation from Logical Forms

Previous works on Natural Language Generation (NLG) from structured data...

Counterfactual Adversarial Learning with Representation Interpolation

Deep learning models exhibit a preference for statistical fitting over l...

Please sign up or login with your details

Forgot password? Click here to reset