DeepAI AI Chat
Log In Sign Up

iNALU: Improved Neural Arithmetic Logic Unit

by   Daniel Schlör, et al.

Neural networks have to capture mathematical relationships in order to learn various tasks. They approximate these relations implicitly and therefore often do not generalize well. The recently proposed Neural Arithmetic Logic Unit (NALU) is a novel neural architecture which is able to explicitly represent the mathematical relationships by the units of the network to learn operations such as summation, subtraction or multiplication. Although NALUs have been shown to perform well on various downstream tasks, an in-depth analysis reveals practical shortcomings by design, such as the inability to multiply or divide negative input values or training stability issues for deeper networks. We address these issues and propose an improved model architecture. We evaluate our model empirically in various settings from learning basic arithmetic operations to more complex functions. Our experiments indicate that our model solves stability issues and outperforms the original NALU model in means of arithmetic precision and convergence.


page 1

page 2

page 3

page 4


Neural Arithmetic Units

Neural networks can approximate complex functions, but they struggle to ...

Neural Power Units

Conventional Neural Networks can approximate simple arithmetic operation...

A Primer for Neural Arithmetic Logic Modules

Neural Arithmetic Logic Modules have become a growing area of interest, ...

Measuring Arithmetic Extrapolation Performance

The Neural Arithmetic Logic Unit (NALU) is a neural network layer that c...

Improving the Robustness of Neural Multiplication Units with Reversible Stochasticity

Multilayer Perceptrons struggle to learn certain simple arithmetic tasks...

Neural Arithmetic Logic Units

Neural networks can learn to represent and manipulate numerical informat...

Neural Programmer: Inducing Latent Programs with Gradient Descent

Deep neural networks have achieved impressive supervised classification ...