Log In Sign Up

End to End Binarized Neural Networks for Text Classification

by   Harshil Jain, et al.

Deep neural networks have demonstrated their superior performance in almost every Natural Language Processing task, however, their increasing complexity raises concerns. In particular, these networks require high expenses on computational hardware, and training budget is a concern for many. Even for a trained network, the inference phase can be too demanding for resource-constrained devices, thus limiting its applicability. The state-of-the-art transformer models are a vivid example. Simplifying the computations performed by a network is one way of relaxing the complexity requirements. In this paper, we propose an end to end binarized neural network architecture for the intent classification task. In order to fully utilize the potential of end to end binarization, both input representations (vector embeddings of tokens statistics) and the classifier are binarized. We demonstrate the efficiency of such architecture on the intent classification of short texts over three datasets and for text classification with a larger dataset. The proposed architecture achieves comparable to the state-of-the-art results on standard intent classification datasets while utilizing   20-40 lesser memory and training time. Furthermore, the individual components of the architecture, such as binarized vector embeddings of documents or binarized classifiers, can be used separately with not necessarily fully binary architectures.


page 2

page 8


Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Most tasks in natural language processing can be cast into question answ...

Rep the Set: Neural Networks for Learning Set Representations

In several domains, data objects can be decomposed into sets of simpler ...

CONSENT: Context Sensitive Transformer for Bold Words Classification

We present CONSENT, a simple yet effective CONtext SENsitive Transformer...

Text Classification for Task-based Source Code Related Questions

There is a key demand to automatically generate code for small tasks for...

JUMPER: Learning When to Make Classification Decisions in Reading

In early years, text classification is typically accomplished by feature...

A character representation enhanced on-device Intent Classification

Intent classification is an important task in natural language understan...