DeepAI AI Chat
Log In Sign Up

Vision-Language Adaptive Mutual Decoder for OOV-STR

09/02/2022
by   Jinshui Hu, et al.
Anhui USTC iFLYTEK Co
0

Recent works have shown huge success of deep learning models for common in vocabulary (IV) scene text recognition. However, in real-world scenarios, out-of-vocabulary (OOV) words are of great importance and SOTA recognition models usually perform poorly on OOV settings. Inspired by the intuition that the learned language prior have limited OOV preformence, we design a framework named Vision Language Adaptive Mutual Decoder (VLAMD) to tackle OOV problems partly. VLAMD consists of three main conponents. Firstly, we build an attention based LSTM decoder with two adaptively merged visual-only modules, yields a vision-language balanced main branch. Secondly, we add an auxiliary query based autoregressive transformer decoding head for common visual and language prior representation learning. Finally, we couple these two designs with bidirectional training for more diverse language modeling, and do mutual sequential decoding to get robuster results. Our approach achieved 70.31% and 59.61% word accuracy on IV+OOV and OOV settings respectively on Cropped Word Recognition Task of OOV-ST Challenge at ECCV 2022 TiE Workshop, where we got 1st place on both settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/19/2022

ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spotting

Scene text spotting is of great importance to the computer vision commun...
08/04/2022

1st Place Solution to ECCV 2022 Challenge on Out of Vocabulary Scene Text Understanding: Cropped Word Recognition

This report presents our winner solution to ECCV 2022 challenge on Out-o...
05/10/2019

Language Modeling with Deep Transformers

We explore multi-layer autoregressive Transformer models in language mod...
05/08/2020

On Vocabulary Reliance in Scene Text Recognition

The pursuit of high performance on public benchmarks has been the drivin...
03/11/2021

Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

Linguistic knowledge is of great benefit to scene text recognition. Howe...