Logic and the 2-Simplicial Transformer

09/02/2019

∙

by James Clift, et al.

∙

The University of Melbourne

∙

We introduce the 2-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

READ FULL TEXT

Logic and the 2-Simplicial Transformer

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

Learning Language Representations with Logical Inductive Bias

Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Logiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoning

Translational Equivariance in Kernelizable Attention

Logic and the 2-Simplicial Transformer

Related Research

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

Learning Language Representations with Logical Inductive Bias

Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Logiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoning

Translational Equivariance in Kernelizable Attention