Dispatcher: A Message-Passing Approach To Language Modelling

05/09/2021
by   Alberto Cetoli, et al.
6

This paper proposes a message-passing mechanism to address language modelling. A new layer type is introduced that aims to substitute self-attention. The system is shown to be competitive with existing methods: Given N tokens, the computational complexity is O(N log N) and the memory complexity is O(N) under reasonable assumptions. In the end, the Dispatcher layer is seen to achieve comparable perplexity to prior results while being more efficient

READ FULL TEXT

page 1

page 2

page 3

page 4

02/01/2022

Memory-based Message Passing: Decoupling the Message for Propogation from Discrimination

Message passing is a fundamental procedure for graph neural networks in ...
08/17/2019

Message Passing Attention Networks for Document Understanding

Most graph neural networks can be described in terms of message passing,...
09/05/2020

A Class of Optimal Structures for Node Computations in Message Passing Algorithms

Consider the computations at a node in the message passing algorithms. A...
06/02/2022

Unified Recurrence Modeling for Video Action Anticipation

Forecasting future events based on evidence of current conditions is an ...
10/24/2020

Pathfinder Discovery Networks for Neural Message Passing

In this work we propose Pathfinder Discovery Networks (PDNs), a method f...
09/17/2020

A type language for message passing component-based systems

Component-based development is challenging in a distributed setting, for...
06/25/2018

A Unified Model with Structured Output for Fashion Images Classification

A picture is worth a thousand words. Albeit a cliché, for the fashion in...

Code Repositories