Logic and the 2-Simplicial Transformer

09/02/2019

∙

We introduce the 2-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

READ FULL TEXT

Logic and the 2-Simplicial Transformer

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

Learning Language Representations with Logical Inductive Bias

Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Transformer in Transformer as Backbone for Deep Reinforcement Learning

Attention Approximates Sparse Distributed Memory

Logic and the 2-Simplicial Transformer

Related Research

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

Learning Language Representations with Logical Inductive Bias

Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Transformer in Transformer as Backbone for Deep Reinforcement Learning

Attention Approximates Sparse Distributed Memory