In the past decades, scene text detection and recognition have drawn considerable attention from the computer vision community, due to its wide applications, e.g, automatic driving, visual auxiliaries , and human-computer interaction . As scene text provides pivotal and specific information, accurate recognition of text plays crucial roles in various real-world scenarios .
, drawing inspiration from neural machine translation, encodes images into features and employs an attention mechanism to align and decode characters. The latter [24, 22], approaching text recognition from a 2D perspective, first adopts a fully convolutional network (FCN) to perform semantic segmentation, then seeks connected components in the segmentation maps, and finally infers the class of each connected component (each is taken as a character).
Essentially, to correctly recognize the content in a cropped text image, the number of characters as well as the order and class of each character should be accurately predicted. RNN-attention based methods usually work well in most cases. However, when there is noise in the background or irregular text shape , the attention mechanism may fail that the center of the estimated attention map targets to a wrong position, causing mistakes in character order and class (see Fig.1). More seriously, due to the recurrent memory mechanism in the RNN module, such errors will accumulate and propagate, making the situation even worse.
Semantic segmentation based algorithms explore a different way and exhibit stronger adaptability to text of different shapes (horizontal, oriented and curved). However, it is difficult to successfully separate each character from the segmentation maps, since improper binarization will result in such embarrassments: one character might be split into multiple parts or multiple characters may stick together (see Fig.1). In these cases, the predictions of number and class of characters would be wrong. In summary, existing approaches, either RNN-attention based or semantic segmentation based, are not able to commendably resolve the difficulties in scene text recognition.
The root cause for the attention drift problem in the RNN-attention based methods might be that the alignment operations (realized with attention maps) rely on both visual features and previous decoding results. Mutual interference might occur between these two types of information. Therefore, it is necessary to perform character alignment and classification in independent branches. Regarding semantic segmentation based algorithms, the assumption that characters can be sought via simple binarization does not hold in challenging scenarios. To address this issue, a natural and feasible solution is to represent the position and order of characters with different channels.
In this paper, we propose a novel text recognition framework, called TextScanner. Like a scanner in the real world, TextScanner can read characters in correct order.
Generally, TextScanner is built upon semantic segmentation . It consists of two branches: one for character classification (class branch) and the other for character position and order prediction (geometry branch) (see Fig. 2
). The class branch produces multi-channel segmentation maps, in which the values at each location represent the probabilities of character classes (including the background class). The geometry branch also produces multi-channel segmentation maps, but the meanings of the values at each location are different from those in the class branch. The characters are sought by multiplying two groups of segmentation maps in an element-wise manner and acquiring the class with maximal probability for each channel. This procedure is termed as word formation in this work.
Since characters are well aligned and the order is ensured, TextScanner can avoid the attention drift phenomenon observed in RNN-attention based methods. Meanwhile, in the geometry branch different characters, even contiguous with each other or with the same class label, are strictly assigned into different channels, so they can be easily extracted.
As FAN  and CA-FCN , TextScanner also requires character level annotations for training, since the geometry branch takes character centers as supervision signals. However, there are actually plenty of real image examples without character level annotations, which could be very beneficial for training text recognizers. To make use of such real data, we devise a mutual supervision mechanism. For image examples without character level annotations, the predictions of the class branch and the geometry branch can supervise each other with only sequence level annotations. In consequence, TextScanner is able to fully utilize all kinds of available training data, including both synthetic and real text images.
We conduct experiments on public benchmarks for scene text recognition to validate the proposed TextScanner. It achieves higher or highly competitive accuracy on regular text datasets and obtains significantly enhanced performance on irregular text datasets. The recognition accuracy increases on ICDAR 2015 and on CUTE80, compared with the previous art. We also evaluate TextScanner on a Chinese recognition task. The quantitative results further prove the superiority of the proposed algorithm. The contributions in this work are summarized as follows:
We propose a novel text recognition framework, which predicts the class and geometry information (position and order) of characters with two separate branches.
We devise a mutual-supervision mechanism, which endows the framework with the ability to make use of both synthetic and real data for training.
The experiments demonstrate that the proposed TextScanner achieves state-of-the-art or highly competitive performance on public benchmarks.
Furthermore, TextScanner exhibits stronger adaptability to longer and more complex text (such as Chinese scripts).
2 Related Work
Text recognition has been a long-standing research topic in computer vision. Research efforts on text recognition can date back to the early age of AI [12, 19]. With the rise of deep learning, scene text recognition has entered a new era. For recent progress in this field, please refer to the survey paper [23, 39]. In this section, we will concentrate on the most relevant works.
Inspired by speech recognition and natural language processing, CTC-based[29, 9] soft-alignment methods and attention-based [7, 31] methods are proposed to handle text recognition as a sequence recognition task. Among them, attention-based methods are prevalent recently, and achieve state-of-the-art performance on public benchmarks. On the other hand, there are still challenging problems in the field of scene text recognition. Text images in natural scenes suffer from the complex background, arbitrary text shape and severe image distortion. Most of current text recognition algorithms are not robust enough to solve hard cases such as text instances which are oriented, curved or extremely blurred.
As stated in Sec. 1, the problem of attention drift is exacerbated by recursive modeling. This problem is also observed in speech recognition , which is where the idea of attention decoder originated from. There are existing works aimed at alleviating this problem. 
proposed to correct attention positions using characters’ class and localization label. New loss function motivated by the formulation of edit distance is presented
to improve the hard alignment of attention decoder. However, these methods do not change the nature of error accumulating which lies in the coupled modeling of attention generation and character classification. Different from these methods, the proposed algorithm uses two separated branches to classify characters and predict the positions and order of characters. The potential mutual interference between alignment and decoding is eliminated and the problem of attention drift can be avoided.
Recently, segmentation-based methods are also introduced to the field of text recognition [24, 22]. Segmentation-based methods are usually more flexible than attention decoders in the recognition of irregular text such as oriented or curved text instances. However, the post-processing of these methods may fail to separate closely arranged characters as shown in Fig. 1. As the characters are recognized by finding and voting inside the connected components in the segmentation map, this restriction limits their recognition accuracy. Besides, the application of segmentation-based methods remains limited because these methods can not use real image examples with only sequence-level annotation. With our proposed method, the characters are naturally separated and ordered by dispatching character localization to different channels. The mutual-supervision mechanism further enables the two branches to utilize sequence level annotations to supervise and enhance each other.
The overall structure of the proposed method is illustrated in Fig. 2. The decoder of the network is composed of two branches: class branch and geometry branch.
The class branch of TextScanner produces character segmentation , which is of resolution and denotes the number of classes (all character classes plus background).
is directly generated from visual features extracted by a CNN backbone. The prediction module is composed of two stacked convolutional layers with kernel sizeand . A Softmax normalization is applied over the class dimension to generate the character segmentation maps.
Firstly, a character localization map
is produced from the same visual features as the class branch, with a Sigmoid activation function. Concurrently, a top-down pyramid structure, in which features in the upsampling path is added by the features with the same resolution from the downsampling path is taken to generate order segmentation maps. Especially, the feature maps in the top layer of the downsampling path are encoded by an RNN (GRU, to be exact) module for context modeling. Following the upsampling path, two convolutional layers are employed to generate the order segmentation maps , where is the pre-defined max sequence length. The order segmentation maps are also normalized by a Softmax operation. Then an order map , which indicates the position of the character in the sequence, can be computed from the channel of order segmentation and the character localization map by an element-wise multiplication:. The detail of the geometry branch is depicted in Fig. 3.
With the produced character clarification maps and order maps , we now present the decoding procedure which formats characters in order. As the order of character locations is encoded into the order maps , the classification scores can be computed from the order maps and character clarification maps as:
is the vector of scores representing the class probabilities of thecharacter. is all valid spatial locations in the space. Similar to attention decoders, once the maximal probability of a character is below a pre-defined threshold or reached the maximal value , the decoding process is terminated.
This decoding procedure is totally differentiable. Therefore, it can be trained within the network using sequence level as well as character level annotations. The optimization process utilizing sequence level annotations is introduced in detail in the following section.
Pre-Training with Character-Level Annotations
When pre-training on synthetic data, TextScanner can be optimized with character-level annotations.
Due to the rectangles are inaccurate in curved or dense text, we keep the definition of character regions polygons , where is the number of points in polygon.
To refrain the overlap caused by edges of adjacent characters, the polygon character bounding box is shrunk to with the Vatti clipping algorithm. Inside area, the class of the corresponding character is rendered as ground truth of the character segmentation. Pixels outside hardly contribute to the decoding of sequences and are ignored in the optimization of text segmentation.
To generate the ground truth of order maps with character-level annotations, the center of Gaussian maps is firstly detected by computing the central points of characters bounding boxes. As Fig. 4 shown,
2D Gaussian maps with and expectation at central points are generated for each character. Then the order of characters is rendered for pixels inside area:
where is the generated order map ground truth for each character, is the order threshold, which set to 0.5 in our experiment. Finally is normalized to , to produce the ground truth of .
Taking the same operation as to , 0-1 normalized Gaussian heatmap can be acquired from . According to all , the ground truth of localization map can be generated by straightforwardly combining heat maps:
The overall loss function is a weighted sum of losses for the three mentioned tasks as
where , , , are the losses for localization map, order segmentation, text segmentation and mutual supervision loss respectively. The detail of mutual supervision loss is illustrated in the next section. In our experiments and are set to 10 for scaling the numerical values. is set to 0 during pretraining otherwise to 1.
The localization map loss is computed as an average smooth l1 loss. The losses for order segmentation() and character segmentation() are computed as cross entropy between the predicted scores and corresponding ground truth. The background class in both segmentation task is ignored in cross entropy computation.
For previous semantic segmentation based methods, it’s critical to obtain accurate locations of all the characters during model training, since the character classification is at pixel-level. This is problematic when character level annotations are not available. To reduce the reliance on character-level annotations, we devise a mutual-supervision mechanism based on the dual-branch structure of TextScanner. As shown in Fig. 2, text sequences can be generated by combining character segmentation maps and order maps . Given a sequence label and one of the two outputs, supervision signals can be generated for the other one.
and are further transformed by taking the index of max value across and channels into and . Given the text sequence label , the mutual-supervision process is carried out from the first character in to the last. For the character in , its order is and the class is . and are coordinates of the pixels that corresponding to the character in and :
Apart from the constraint of class or order, we add the constraint of to keep the attended regions in the center of characters. is set to 0.2 in our experiment.
For the mutual supervision purpose, we use , which is derived from ,in the supervision of and , which is derived from , in the supervision of :
is the cross entropy loss and
is the one hot encoding function. The step 1 of cross supervision process is shown in Fig5(a). The process is carried on till the last character of . Note that multiple regions of are chosen in one selection for characters occur more than once in and can’t be used in supervision of , as shown in Fig 5(b). So we remove these samples from the cross supervision process. The confidences for and are denoted as and :
For the character in , if the number of pixels in its corresponding region in or is larger than 0, the character is considered exists in or . is the length of . The more the characters in exists, the higher is. when all the characters exists in the prediction map. and are included in the loss computation of the text sequence:
In our experiment, is set to 0.2, is set to 2 to further reduce the impact of inaccurate predictions. After being pre-trained on synthetic datasets with character level annotations, our model can be further fine-tuned on real-world datasets or synthetic datasets with only sequence level annotation by adopting this cross supervision mechanism, which is impossible in previous segmentation-based methods.
Our model is built on top of the backbone from CA-FCN, in which the character attentions are removed and VGG blocks are replaced with a ResNet-50 base model. The score threshold is set as 0.3 empirically, and the max size is set as 32 in our implementations.
|Almazán et al.||-||91.2||82.1||-||89.2||-||-||-||-||-|
|Jaderberg et al.||-||-||-||86.1||-||-||-||-||-||-|
|Su et al.||-||-||-||-||83.0||-||-||-||-||-|
|Jaderberg et al.||90k||97.1||92.7||-||95.4||80.7||90.8||-||-||-|
|Jaderberg et al.||90k||95.5||89.6||-||93.2||71.7||81.8||-||-||-|
|Yang et al.||90k||97.8||96.1||-||95.2||-||-||-||75.8||69.3|
|ST + 90k||99.3||97.5||87.4||97.1||85.9||93.3||70.6||-||-|
|||ST + 90k||99.6||98.1||87.0||96.0||82.8||-||68.2||73.0||76.8|
|||ST + 90k||99.5||97.9||88.3||96.6||87.5||94.4||73.9||-||-|
|||ST + 90k||99.6||98.8||93.4||97.4||89.5||91.8||76.1||78.5||79.5|
|TextScanner+90k||ST + 90k||99.7||99.1||93.9||98.5||90.1||92.9||79.4||84.3||83.3|
|SAR||ST + 90k + real||99.4||98.2||95.0||98.5||91.2||94.0||78.8||86.4||89.6|
|TextScanner+real||ST + 90k + real||99.8||99.5||95.7||99.4||92.7||94.9||83.5||84.8||91.6|
” use the training set of real data. “0”, “50” and “1k” indicate the size of the lexicons, “0” means no lexicon.
We conduct experiments on standard benchmarks to evaluate TextScanner and compare it with other competitors.
ICDAR 2013(IC13)  recognition task provides 288 scene images with annotations, from which 1015 word images are cropped. Besides, the dataset provides character-level bounding boxes.
ICDAR 2015(IC15)  consists of 1000 images with word-level quadrangles annotation for training and 500 for testing.
IIIT 5K-Words(IIIT)  dataset contains 5K word images for scene text recognition.
Street View Text(SVT)  dataset has 350 images and only word-level annotations are provided.
SVT-Perspective(SVTP)  dataset contains 639 cropped images for testing. Many images in the dataset are heavily distorted.
CUTE80(CT)  dataset is taken in natural scene. It consists of 80 high-resolution images with no lexicon.
ICDAR 2017 MLT(MLT-2017)  is comprised of 9000 training images and 9000 test images. We acquire cropped word instances for recognition by using the quadrilateral word-level annotation.
SynthText  consists of 80k images for training. We cropped about 7 million instances with character and word-level bounding-boxes annotations from the training set.
Synth90k  contains 8 millions word images from 90k English words with word-level annotation.
In addition, approximately 16k real images are collected from the training sets released by the mentioned datasets for fine-tuning.
The training procedure of TextScanner includes two stages: we firstly use the synthetic dataset SynthText with character-level annotations to pre-train the model for epochs, then the real image examples with sequence-level annotations are mixed into the training set for fine-tuning the network for more epoch. Our methods in Tab.1 use different data for training, “TextScanner-pre” and “TextScanner-mutual” are the models trained on SynthText with and without mutual-supervision, respectively; “TextScanner+90k” and “TextScanner+real” are fine-tuned with the 90k dataset and the training set of real data.
We use Adam optimizer for training of all experiments. The learning rate is initialized as and the decays to and . During training and inference, the input images are resized to .
Recognition Performance Evaluation
The recognition accuracies of different methods on standard benchmarks, including regular (IIIT, SVT, IC13) and irregular (IC15, SVTP, CT) text, are shown in Tab. 1.
The natural modeling of TextScanner makes it more robust to hard cases where the text is curved or oriented. The three variants of TextScanner in Tab. 1 consistently outperform previous methods in comparison with the same training data. Especially on curved text, “TextScanner+90k”, which is trained with synthetic data, achieves an improvement of on IC15, on SVTP, and on CT. The advantages of TextScanner stem from aspects: (1) It is segmentation based, which makes the prediction more relevant to visual features and free from error accumulation brought by the recursive modeling. (2) It scans characters one by one and ensures they are read in the right order and separated properly. Some recognition examples are visualized in Fig. 6.
“TextScanner+real” brings an even more significant boost in performance and demonstrates the capability of TextScanner to utilize real data for training, which also verifies the effectiveness of the proposed mutual-supervision mechanism. Moreover, although the mutual-supervision mechanism is designed for making use of real-world data with sequence-level annotations, fine-tuning with synthetic SynthText and 90k datasets consistently bring performance improvement.
Chinese Recognition Evaluation
Preceding experiments are based on datasets with English text, whose alphabet is relatively small. To further validate the capability of TextScanner, we conduct experiments on a Chinese recognition task, which is more challenging due to its larger alphabet and more complex visual variations of characters. We compare the performance of TextScanner in Chinese recognition with two representative methods, CRNN and ASTER
. We use their open-source implementations for comparison. The models are trained with the same training data, which is generated by the synthetic engine released with SynthText and evaluated on cropped text images from the validation set of MLT-17.
where is the number of text instances.
As shown in Tab. 2, TextScanner outperforms attention decoder with a large margin, in recognition accuracy and in normalized edit distance. The results demonstrate that TextScanner can handle such challenging recognition tasks better.
The main reason for this improvement is that TextScanner separate geometry branch from the class branch. For the Chinese recognition task, models are more prone to classification errors due to the much more complex structures and appearances of Chinese characters.
In contrast to attention decoders, the dual-branch architecture of TextScanner is more robust to the problem of error accumulating towards difficulty in classification. The class branch and the geometry branch are optimized individually in pre-training, therefore the extraction of character orders is not affected by the probable errors in classification. In the fine-tuning stage, the accurate order extraction can improve the class branch in return through the mutual-supervision mechanism.
Character Localization Accuracy of TextScanner
For both attention decoder and TextScanner, accurate prediction of attention position or character localization is fundamental for recognition. As they both produce the center of characters, we compare their performance in character localization on the IC13 dataset. As IC13 provides annotations of character positions in the image, the two methods are evaluated by measuring the normalized distance between the produced character center and ground truth center position in the width axis.
The probability density of center distance in IC13 is illustrated in Fig.7. The probability of TextScanner to have accurate localization() is obviously greater than attention decoder. This proves that TextScanner gives more accurate character localization results.
5 Ablation Study
The superiority of TextScanner has been validated via the experiments in Chinese recognition task, here we specifically explore the effect of the geometry branch and the word formation process based on order maps. In Tab.3, the geometry branch and word formation are ablated respectively. Note that the word formation with order maps relies on the output of the geometry branch, therefore we use the post-processing procedure of CA-FCN as a replacement.
The experimental results clearly show the improvements brought by the geometry branch and its decoding process (the second and the third row in Tab.3). As the order maps ensure the characters are scanned in correct order, the recognition performance is significantly elevated, especially on irregular datasets: on IC15 and on SVTP. Besides, even with the regular post-processing, the geometry branch still achieves better performance, proving it can facilitate the optimization of the class branch.
In this paper, we have presented TextScanner, an effective segmentation-based dual-branch framework for scene text recognition. TextScanner can overcome the problems and defects of previous methods, and work well under various challenging scenarios. A novel mutual-supervision mechanism, which makes it possible to take full advantage of both real and synthetic data, is also proposed. Besides, TextScanner shows stronger adaptability in handling difficult text.
This work was partly supported by National Natural Science Foundation of China (61733007).
-  (2014) Word spotting and recognition with embedded attributes. PAMI. Cited by: Table 1.
-  (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Cited by: §1.
-  (2018) Edit probability for scene text recognition. In CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 1508–1516. External Links: Cited by: §2, Table 1.
-  (2017-10) Focusing attention: towards accurate text recognition in natural images. In ICCV 2017, pp. 5086–5094. External Links: Cited by: Figure 1, §1, §2, Table 1.
-  (2018) AON: towards arbitrarily-oriented text recognition. In CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 5571–5579. External Links: Cited by: §1, Table 1.
Gated feedback recurrent neural networks. In
International Conference on Machine Learning, pp. 2067–2075. Cited by: §3.
Visual attention models for scene text recognition. In 2017 14th IAPR ICDAR, Vol. 01, pp. 943–948. External Links: Cited by: §1, §2.
-  (2015) Supervised mid-level features for word image representation. In CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 2956–2964. External Links: Cited by: Table 1.
-  (2006) Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine learning, Pittsburgh, Pennsylvania, USA, pp. 369–376. Cited by: §1, §2.
-  (2016) Synthetic data for text localisation in natural images. In CVPR, pp. 2315–2324. Cited by: §4.
-  (2016) Deep residual learning for image recognition. In PAMI, pp. 770–778. Cited by: §3.
-  (1982) The history of ocr, optical character recognition. Manchester Center, VT: Recognition Technologies Users Association. Cited by: §2.
-  (2014) Deep structured output learning for unconstrained text recognition. arXiv preprint arXiv:1412.5903. Cited by: Table 1.
-  (2014) Synthetic data and artificial neural networks for natural scene text recognition. NIPS Deep Learning Workshop. Cited by: §4.
Reading text in the wild with convolutional neural networks. International Journal of Computer Vision 116 (1), pp. 1–20. Cited by: Table 1.
-  (2013-08) ICDAR 2013 robust reading competition. In 2013 12th ICDAR, pp. 1484–1493. External Links: Cited by: §4.
-  (2015) ICDAR 2015 competition on robust reading. In 2015 13th ICDAR, pp. 1156–1160. Cited by: §4.
-  (2017) Joint ctc-attention based end-to-end speech recognition using multi-task learning. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4835–4839. Cited by: §2.
-  (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), pp. 2278–2324. Cited by: §2.
-  (2016) Recursive recurrent nets with attention modeling for ocr in the wild. In CVPR, pp. 2231–2239. Cited by: Table 1.
-  (2019) Show, attend and read: a simple and strong baseline for irregular text recognition. In AAAI, Vol. 33, pp. 8610–8617. Cited by: Table 1.
-  (2019) Scene text recognition from two-dimensional perspective. In AAAI, Cited by: §1, §1, §1, §2, Table 1.
-  (2018) Scene text detection and recognition: the deep learning era. arXiv preprint arXiv:1811.04256. Cited by: §2.
-  (2018) Mask textspotter: an end-to-end trainable neural network for spotting text with arbitrary shapes. In ECCV, pp. 67–83. Cited by: §1, §2.
-  (2012) Scene text recognition using higher order language priors. In BMVC-British Machine Vision Conference, Cited by: §4.
-  (2017-11) ICDAR2017 robust reading challenge on multi-lingual scene text detection and script identification - rrc-mlt. In 2017 14th IAPR ICDAR, Vol. 01, pp. 1454–1459. External Links: Cited by: §4.
-  (2013-12) Recognizing text with perspective distortion in natural scenes. In 2013 IEEE International Conference on Computer Vision, Vol. , pp. 569–576. External Links: Cited by: §1, §4.
-  (2014) A robust arbitrary text detection system for natural scene images. Expert Systems with Applications 41 (18), pp. 8027 – 8048. External Links: Cited by: §4.
-  (2017) An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. PAMI 39 (11), pp. 2298–2304. Cited by: §2, §4, Table 1, Table 2.
-  (2016) Robust scene text recognition with automatic rectification. In CVPR, pp. 4168–4176. Cited by: §1, Table 1.
-  (2018) ASTER: an and attentional scene and text recognizer and with flexible and rectification. In PAMI, Vol. , pp. 1–1. External Links: Cited by: §2, §4, Table 1, Table 2.
-  (2017) Icdar2017 competition on reading chinese text in the wild (rctw-17). In 2017 14th IAPR ICDAR, Vol. 1, pp. 1429–1434. Cited by: §4.
-  (2015) Accurate scene text recognition based on recurrent neural network. In Computer Vision – ACCV 2014, D. Cremers, I. Reid, H. Saito, and M. Yang (Eds.), Cham, pp. 35–48. External Links: Cited by: Table 1.
-  (1992) A generic solution to polygon clipping. Communications of the ACM 35 (7), pp. 56–64. Cited by: §3.
-  (2011-11) End-to-end scene text recognition. In ICCV, ICCV ’11, Washington, DC, USA, pp. 1457–1464. External Links: Cited by: §4.
-  (2012-11) End-to-end text recognition with convolutional neural networks. In ICPR2012, Vol. , pp. 3304–3308. External Links: Cited by: §1.
Learning to read irregular text with attention mechanisms.
Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 3280–3286. Cited by: Table 1.
-  (2014) Strokelets: a learned multi-scale representation for scene text recognition. In CVPR, pp. 4042–4049. Cited by: Table 1.
-  (2016) Scene text detection and recognition: recent advances and future trends. Frontiers of Computer Science 10 (1), pp. 19–36. Cited by: §2.