iNLTK: Natural Language Toolkit for Indic Languages

09/26/2020
by   Gaurav Arora, et al.
0

We present iNLTK, an open-source NLP library consisting of pre-trained language models and out-of-the-box support for Paraphrase Generation, Textual Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text Generation in 13 Indic Languages. By using pre-trained models from iNLTK for text classification on publicly available datasets, we significantly outperform previously reported results. On these datasets, we also show that by using pre-trained models and paraphrases from iNLTK, we can achieve more than 95 the previous best performance by using less than 10 iNLTK is already being widely used by the community and has 40,000+ downloads, 600+ stars and 100+ forks on GitHub. The library is available at https://github.com/goru001/inltk.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/30/2020

AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages

We present the IndicNLP corpus, a large-scale, general-domain corpus con...
research
11/26/2019

Word-Class Embeddings for Multiclass Text Classification

Pre-trained word embeddings encode general word semantics and lexical re...
research
05/16/2023

Pre-Training to Learn in Context

In-context learning, where pre-trained language models learn to perform ...
research
02/08/2023

GPTScore: Evaluate as You Desire

Generative Artificial Intelligence (AI) has enabled the development of s...
research
11/18/2022

GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation

We introduce GENIUS: a conditional text generation model using sketches ...
research
02/15/2022

textless-lib: a Library for Textless Spoken Language Processing

Textless spoken language processing research aims to extend the applicab...
research
05/04/2023

HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec

Audio codec models are widely used in audio communication as a crucial t...

Please sign up or login with your details

Forgot password? Click here to reset