compare-mt: A Tool for Holistic Comparison of Language Generation Systems

1 Introduction

Tasks involving the generation of natural language are ubiquitous in NLP, including machine translation (MT; koehn10smt), language generation from structured data Reiter and Dale (2000), summarization Mani (1999), dialog response generation Oh and Rudnicky (2000), image captioning Mitchell et al. (2012). Unlike tasks that involve prediction of a single label such as text classification, natural language texts are nuanced, and there are not clear yes/no distinctions about whether outputs are correct or not. Evaluation measures such as BLEU Papineni et al. (2002), ROUGE Lin (2004), METEOR Denkowski and Lavie (2011), and many others attempt to give an overall idea of system performance, and technical research often attempts to improve accuracy according to these metrics.

However, as useful as these metrics are, they are often opaque: if we see, for example, that an MT model has achieved a gain in one BLEU point, this does not tell us what characteristics of the output have changed. Without fine-grained analysis, readers of research papers, or even the writers themselves can be left scratching their heads asking “what exactly is the source of the gains in accuracy that we’re seeing?”

Figure 1: Workflow of using compare-mt for analysis of two systems

Unfortunately, this analysis can be time-consuming and difficult. Manual inspection of individual examples can be informative, but finding salient patterns for unusual phenomena requires perusing a large number of examples. There is also a risk that confirmation bias will simply affirm pre-existing assumptions. If a developer has some hypothesis about specifically what phenomena their method should be helping with, they can develop scripts to automatically test these assumptions. However, this requires deep intuitions with respect to what changes to expect in advance, which cannot be taken for granted in beginning researchers or others not intimately familiar with the task at hand. In addition, creation of special-purpose one-off analysis scripts is time-consuming.

In this paper, we present compare-mt, a tool for holistic comparison and analysis of the results of language generation systems. The main use case of compare-mt, illustrated in 1, is that once a developer obtains multiple system outputs (e.g. from a baseline system and improved system), they feed these outputs along with a reference output into compare-mt, which extracts aggregate statistics comparing various aspects of these outputs. The developer can then quickly browse through this holistic report and note salient differences between the systems, which will then guide fine-grained analysis of specific examples that elucidate exactly what is changing between the two systems.

Examples of the aggregate statistics generated by compare-mt are shown in §2, along with description of how these lead to discovery of salient differences between systems. These statistics include word-level accuracies for words of different types, sentence-level accuracies or counts for sentences of different types, and salient $n$ -grams or sentences where one system does better than the other. §4 demonstrates compare-mt’s practical applicability by showing some case studies where has already been used for analysis in our previously published work. §3 further details more advanced functionality of compare-mt that can make use of specific labels, perform analysis over source side text through alignments, and allow simple extension to new types of analysis. The methodology in compare-mt is inspired by several previous works on automatic error analysis Popović and Ney (2011), and we perform an extensive survey of the literature, note how many of the methods proposed in previous work can be easily realized by using functionality in compare-mt, and detail the differences with other existing toolkits in §5.

2 Basic Analysis using compare-mt

Using compare-mt with the default settings is as simple as typing

compare-mt ref sys1 sys2

where ref is a manually curated reference file, and sys1 and sys2 are the outputs of two systems that we would like to compare. These analysis results can be written to the terminal in text format, but can also be written to a formatted HTML file with charts and LaTeX tables that can be directly used in papers or reports.²²2In fact, all of the figures and tables in this paper (with the exception of Fig. 1) were generated by compare-mt, and only slightly modified for formatting. An example of the command used to do so is shown in the Appendix.

In this section, we demonstrate the types of analysis that are provided by this standard usage of compare-mt. Specifically, we use the example of comparing phrase-based Koehn et al. (2003) and neural Bahdanau et al. (2015) Slovak-English machine translation systems from neubig18emnlp.

	PBMT	NMT	Win?
BLEU	22.43	24.03	s2 $>$ s1
	[21.76,23.19]	[23.33,24.65]	p $<$ 0.001
RIBES	80.00	80.00	-
	[79.39,80.64]	[79.44,80.92]	p=0.44
Length	94.79	93.82	s1 $>$ s2
	[94.10,95.49]	[92.90,94.85]	p $<$ 0.001

Table 1:

Aggregate score analysis with scores, confidence intervals, and pairwise significance tests.

Figure 2: Analysis of word F-measure bucketed by frequency in the training set.

Figure 3: BLEU scores bucketed by sentence length.

Figure 4: Counts of sentences by length difference between the reference and the output.

Figure 5: Counts of sentences by sentence-level BLEU bucket.

Aggregate Score Analysis

The first variety of analysis is not unique to compare-mt, answering the standard question posed by most research papers: “given two systems, which one has better accuracy overall?” It can calculate scores according to standard BLEU Papineni et al. (2002)

, as well as other measures such as output-to-reference length ratio (which can discover systematic biases towards generating too-long or too-short sentences) or alternative evaluation metrics such as chrF

Popović (2015) and RIBES Isozaki et al. (2010). compare-mt also has an extensible Scorer class, which will be used to expand the metrics supported by compare-mt in the future, and can be used by users to implement their own metrics as well. Confidence intervals and significance of differences in these scores can be measured using bootstrap resampling Koehn (2004).

Fig. 1 shows the concrete results of this analysis on our PBMT and NMT systems. From the results we can see that the NMT achieves higher BLEU but shorter sentence length, while there is no significant difference in RIBES.

Bucketed Analysis

A second, and more nuanced, variety of analysis supported by compare-mt is bucketed analysis, which assigns words or sentences to buckets, and calculates salient statistics over these buckets.

Specifically, bucketed word accuracy analysis attempts to answer the question “which types of words can each system generate better than the other?” by calculating word accuracy by bucket. One example of this, shown in Fig. 2, is measurement of word accuracy bucketed by frequency in the training corpus. By default this “accuracy” is defined as f-measure of system outputs with respect to the reference, which gives a good overall picture of how well the system is doing, but it is also possible to separately measure precision or recall, which can demonstrate how much a system over- or under-produces words of a specific type as well. From the results in the example, we can see that both PBMT and NMT systems do more poorly on rare words, but the PBMT system tends to be more robust to low-frequency words while the NMT system does a bit better on very high-frequency words.

A similar analysis can be done on the sentence level, attempting to answer questions of “on what types of sentences can one system perform better than the other?” In this analysis we define the “bucket type”, which determines how we split sentences into bucket, and the “statistic” that we calculate for each of these buckets. For example, compare-mt calculates three types of analysis by default:

bucket=length, statistic=score: This calculates the BLEU score by reference sentence length, indicating whether a system does better or worse at shorter or longer sentences. From the Fig. 3, we can see that the PBMT system does better at very long sentences, while the NMT system does better at very short sentences.
bucket=lengthdiff, statistic=count: This outputs a histogram of the number of sentences that have a particular length difference from the reference output. A distribution peaked around 0 indicates that a system generally matches the output length, while a flatter distribution indicates a system has trouble generating sentences of the correct length Fig. 4 indicates that while PBMT rarely generates extremely short sentences, NMT has a tendency to do so in some cases.
bucket=score, statistic=count: This outputs a histogram of the number of sentences receiving a particular score (e.g. sentence-level BLEU score). This shows how many sentences of a particular accuracy each system outputs. Fig. 5, we can see that the PBMT system has slightly more sentences with low scores.

These are just three examples of the many different types of sentence-level analysis that are possible with difference settings of the bucket and statistic types.

N-gram Difference Analysis

The holistic analysis above is quite useful when word or sentence buckets can uncover salient accuracy differences between the systems. However, it is also common that we may not be able to predict a-priori what kinds of differences we might expect between two systems. As a method for more fine-grained analysis, compare-mt implements a method that looks at differences in the $n$ -grams produced by each system, and tries to find $n$ -grams that each system is better at producing than the other Akabe et al. (2014). Specifically, it counts the number of times each system matches each ngram $x$ , defined as $m_{1} (x)$ and $m_{2} (x)$

respectively, and calculates a smoothed probability of an

n

-gram match coming from one system or another:

p (x) = \frac{m_{1} (x) + α}{m_{1} (x) + m_{2} (x) + 2 α} .

(1)

Intuitively, $n$ -grams where the first system excels will have a high value (close to 1), and when the second excels the value will be low (close to 0). If smoothing coefficient $α$ is set high, the system will prefer frequent $n$ -grams with robust statistcs, and when $α$ is low, the system will prefer highly characteristic $n$ -grams with a high ratio of matches in one system compared to the other.

$n$ -gram	$m_{1}$	$m_{2}$	$s$
phantom	34	1	0.945
Amy	9	0	0.909
, who	8	0	0.900
my mother	7	0	0.889
else happened	5	0	0.857
going to show you	0	6	0.125
going to show	0	6	0.125
hemisphere	0	5	0.143
Is	0	5	0.143
’m going to show	0	5	0.143

Table 2: Examples discovered by

n

-gram analysis

An example of $n$ -grams discovered with this analysis is shown in Tab. 2. From this, we can then explore the references and outputs of each system, and figure out what phenomena resulted in these differences in $n$ -gram accuracy. For example, further analysis showed that the relatively high accuracy of “hemisphere” for the NMT system was due to the propensity of the PBMT system to output the mis-spelling “hemispher,” which it picked up from a mistaken alignment. This may indicate the necessity to improve alignments for word stems, a problem that could not have easily been discovered from the bucketed analysis in the previous section.

Sentence Example Analysis

Finally, compare-mt makes it possible to analyze and compare individual sentence examples based on statistics, or differences of statistics. Specifically, we can calculate a measure of accuracy of each sentence (e.g. sentence-level BLEU score), sort the sentences in the test set according to the difference in this measure, then display the examples where the difference in evaluation is largest in either direction.

Ref/Sys	BLEU	Text
Ref	-	Beth Israel ’s in Boston .
PBMT	1.00	Beth Israel ’s in Boston .
NMT	0.41	Beat Isaill is in Boston .
Ref	-	And what I ’m talking about is this .
PBMT	0.35	And that ’s what I ’m saying is this .
NMT	1.00	And what I ’m talking about is this .

Table 3: Sentence-by-sentence examples

Tab. 3 shows two examples (cherry-picked from the top 10 sentence examples due to space limitations). We can see that in the first example, the PBMT-based system performs better on accurately translating a low-frequency named entity, while in the second example the NMT system accurately generates a multi-word expression with many frequent words. These concrete examples can both help reinforce our understanding of the patterns found in the holistic analysis above, or uncover new examples that may lead to new methods for holistic analysis.

3 Advanced Features

Here we discuss advanced features that allow for more sophisticated types of analysis using other sources of information than the references and system outputs themselves.

Label-wise Abstraction

One feature that greatly improves the flexibility of analysis is compare-mt’s ability to do analysis over arbitrary word labels. For example, we can perform word accuracy analysis where we bucket the words by POS tags, as shown in 6. In the case of the PBMT vs. NMT analysis above, this uncovers the interesting fact that PBMT was better at generating base-form verbs, whereas NMT was better at generating conjugated verbs. This can also be applied to the $n$ -gram analysis, finding which POS $n$ -grams are generated well by one system or another, a type of analysis that was performed by chiang05hiero to understand differences in reordering between different systems.

Labels are provided by external files, where there is one label per word in the reference and system outputs, which means that generating these labels can be an arbitrary pre-processing step performed by the user without any direct modifications to the compare-mt code itself. These labels do not have to be POS tags, of course, and can also be used for other kinds of analysis. For example, one may perform analysis to find accuracy of generation of words with particular morphological tags Popović et al. (2006)

, or words that appear in a sentiment lexicon

Mohammad et al. (2016).

Source-side Analysis

While most analysis up until this point focused on whether a particular word on the target side is accurate or not, it is also of interest what source-side words are or are not accurately translated. compare-mt also supports word accuracy analysis for source-language words given the source language input file, and alignments between the input, and both the reference and the system outputs. Using alignments, compare-mt finds what words on the source side were generated correctly or incorrectly on the target side, and can do aggregate word accuracy analysis, either using word frequency or labels such as POS tags.

Figure 6: Analysis of word F-measure bucketed by POS tag.

Word Likelihood Analysis

Finally, as many recent methods can directly calculate a log likelihood for each word, we also provide another tool compare-ll that makes it possible to perform holistic analysis of these log likelihoods. First, the user creates a file where there is one log likelihood for each word in the reference file, and then, like the word accuracy analysis above, compare-ll can calculate aggregate statistics for this log likelihood based on word buckets.

Extending compare-mt

One other useful feature is compare-mt’s ability to be easily extended to new types of analysis. For example,

If a user is interested in using a different evaluation metric, they could implement a new instance of the Scorer class and use it for both aggregate score analysis (with significance tests), sentence bucket analysis, or sentence example analysis.
If a user wanted to bucket words according to a different type of statistic or feature, they could implement their own instance of a Bucketer class, and use this in the word accuracy analysis.

4 Example Use-cases

Over the past year or so, we have already been using compare-mt in our research to accelerate the analysis of our results and figure out what directions are most promising to pursue next. Accordingly, results from compare-mt have already made it into a number of our published papers. For example:

Figs. 4 and 5 of wang18emnlptrdec can be generated using sentence bucket analysis to measure “bucket=length, statistic=score” and “bucket=lengthdiff, statistic=count”.
Tab. 7 of qi18naacl shows the results of $n$ -gram analysis, and Fig. 2 shows the results of frequency-based word accuracy analysis.
Fig. 4 of sachan18wmt shows the results of frequency-based word accuracy analysis.
Tab. 8 of michel18emnlp used compare-mt to compare under and over-generated n-grams.
Tab. 5 of kumar2018vmf used compare-mt for frequency-based word accuracy analysis.

5 Related Research and Tools

There have been a wide variety of tools and methods developed to perform analysis of machine translation results. These can be broadly split into those that attempt to perform holistic analysis and those that attempt to perform example-by-example anaylsis.

compare-mt is a tool for holistic analysis over the entire corpus, and many of the individual pieces of functionality provided by compare-mt are inspired by previous works on this topic. Our word error rate analysis is inspired by previous work on automatic error analysis, which takes a typology of errors Flanagan (1994); Murata et al. (2005); Vilar et al. (2006), and attempts to automatically predict which sentences contain these errors Popović and Ney (2011); Zeman et al. (2011); Fishel et al. (2012). Many of the ideas contained in these works can be used easily in compare-mt. Measuring word matches, insertions, and deletions decomposed over POS/morphological tags Popović et al. (2006); Popović and Ney (2007); Zeman et al. (2011); El Kholy and Habash (2011) or other “linguistic checkpoints” Zhou et al. (2008); Naskar et al. (2011) can be largely implemented using the labeled bucketing functionality described in §3. Analysis of word reordering accuracy Birch et al. (2010); Popović and Ney (2011); Bentivogli et al. (2016) can be done through the use of reordering-sensitive measures such as RIBES as described in §2. In addition, the extraction of salient $n$ -grams is inspired by similar approaches for POS $n$ -gram Chiang et al. (2005); Lopez and Resnik (2005) and word $n$ -gram Akabe et al. (2014) based analysis respectively. To the best of our knowledge, and somewhat surprisingly, no previous analysis tool has included the flexible sentence-bucketed analysis that is provided by compare-mt.

One other practical advantage of compare-mt compared to other tools is that it is publicly available under the BSD license on GitHub,³³3https://github.com/neulab/compare-mt and written in modern Python, which is quickly becoming the standard program language of the research community. Many other tools are either no longer available Stymne (2011), or written in other languages such as Perl Zeman et al. (2011) or Java Naskar et al. (2011), which provides some degree of practical barrier to their use and extension.

In contrast to the holistic methods listed above, example-by-example analysis methods attempt to intelligently visualize single translation outputs in a way that can highlight salient differences between the outputs of multiple systems, or understand the inner workings of a system. There are a plethora of tools that attempt to make the manual analysis of individual outputs of multiple systems, through visualization or highlighting of salient parts of the output Lopez and Resnik (2005); Stymne (2011); Zeman et al. (2011); Madnani (2011); Aziz et al. (2012); Gonzàlez et al. (2012); Federmann (2012); Chatzitheodorou and Chatzistamatis (2013); Klejch et al. (2015). There has also been work that attempts to analyze the internal representations or alignments of phrase-based DeNeefe et al. (2005); Weese and Callison-Burch (2010) and neural Ding et al. (2017); Lee et al. (2017) machine translation systems to attempt to understand why they arrived at the decisions they did. While these tools are informative, they play a complementary role to the holistic analysis that compare-mt proposes, and adding the ability to more visually examine individual examples to compare-mt in a more extensive manner is planned as future work.

Recently, there has been a move towards creating special-purpose diagnostic test sets designed specifically to test an MT system’s ability to handle a particular phenomenon. For example, these cover things like grammatical agreement Sennrich (2017), translation of pronouns or other discourse-sensitive phenomena Müller et al. (2018); Bawden et al. (2018), or diagnostic tests for a variety of different phenomena Isabelle et al. (2017). These sets are particularly good for evaluating long-tail phenomena that may not appear in naturalistic data, but are often limited to specific language pairs and domains. compare-mt takes a different approach of analyzing the results on existing test sets and attempting to extract salient phenomena, an approach that affords more flexibilty but less focus than these special-purpose methods.

6 Conclusion

In this paper, we presented an open-source tool for holistic analysis of the results of machine translation or other language generation systems. It makes it possible to discover salient patterns that may help guide further analysis.

compare-mt is evolving, and we plan to add more functionality as it becomes necessary to further understand cutting-edge techniques for MT. One concrete future plan includes better integration with example-by-example analysis (after doing holistic analysis, clicking through to individual examples that highlight each trait), but many more improvements will be made as the need arises.

Acknowledgements: The authors thank the early users of compare-mt and anonymous reviewers for their feedback and suggestions (especially Reviewer 1, who found a mistake in a figure!). This work is sponsored in part by Defense Advanced Research Projects Agency Information Innovation Office (I2O) Program: Low Resource Languages for Emergent Incidents (LORELEI) under Contract No. HR0011-15-C0114. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

References

Akabe et al. (2014) Koichi Akabe, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Discriminative language models as a tool for machine translation error analysis. In Proc. COLING, pages 1124–1132.
Aziz et al. (2012) Wilker Aziz, Sheila Castilho, and Lucia Specia. 2012. Pet: a tool for post-editing and assessing machine translation. In Proc. LREC, pages 3982–3987.
Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR.
Bawden et al. (2018) Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In Proc. NAACL, pages 1304–1313, New Orleans, Louisiana. Association for Computational Linguistics.
Bentivogli et al. (2016) Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proc. EMNLP, pages 257–267, Austin, Texas. Association for Computational Linguistics.
Birch et al. (2010) Alexandra Birch, Miles Osborne, and Phil Blunsom. 2010. Metrics for mt evaluation: evaluating reordering. Machine Translation, 24(1):15–26.
Chatzitheodorou and Chatzistamatis (2013) Konstantinos Chatzitheodorou and Stamatis Chatzistamatis. 2013. COSTA MT evaluation tool: An open toolkit for human machine translation evaluation. The Prague Bulletin of Mathematical Linguistics, 100(1):83–89.
Chiang et al. (2005) David Chiang, Adam Lopez, Nitin Madnani, Christof Monz, Philip Resnik, and Michael Subotin. 2005. The hiero machine translation system: Extensions, evaluation, and analysis. In Proc. EMNLP, pages 779–786.
DeNeefe et al. (2005) Steve DeNeefe, Kevin Knight, and Hayward H. Chan. 2005. Interactively exploring a machine translation model. In Proc. ACL, pages 97–100. Association for Computational Linguistics.
Denkowski and Lavie (2011) Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proc. WMT, pages 85–91.
Ding et al. (2017) Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and understanding neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1150–1159.
El Kholy and Habash (2011) Ahmed El Kholy and Nizar Habash. 2011. Automatic error analysis for morphologically rich languages. In Proc. MT Summit, pages 225–232.
Federmann (2012) Christian Federmann. 2012. Appraise: an open-source toolkit for manual evaluation of mt output. The Prague Bulletin of Mathematical Linguistics, 98(1):25–35.
Fishel et al. (2012) Mark Fishel, Rico Sennrich, Maja Popović, and Ondřej Bojar. 2012. Terrorcat: a translation error categorization-based mt quality metric. In Proc. WMT, pages 64–70.
Flanagan (1994) Mary Flanagan. 1994. Error classification for mt evaluation. In Proc. AMTA, pages 65–72.
Gonzàlez et al. (2012) Meritxell Gonzàlez, Jesús Giménez, and Lluís Màrquez. 2012. A graphical interface for mt evaluation and error analysis. In Proceedings of the ACL 2012 System Demonstrations, pages 139–144.
Isabelle et al. (2017) Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proc. EMNLP, pages 2476–2486.
Isozaki et al. (2010) Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proc. EMNLP, pages 944–952.
Klejch et al. (2015) Ondřej Klejch, Eleftherios Avramidis, Aljoscha Burchardt, and Martin Popel. 2015. Mt-compareval: Graphical evaluation interface for machine translation development. The Prague Bulletin of Mathematical Linguistics, 104(1):63–74.
Koehn (2004) Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. EMNLP, pages 388–395.
Koehn (2010) Philipp Koehn. 2010. Statistical Machine Translation. Cambridge Press.
Koehn et al. (2003) Phillip Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. HLT, pages 48–54.
Kumar and Tsvetkov (2019) Sachin Kumar and Yulia Tsvetkov. 2019. Von mises-fisher loss for training sequence to sequence models with continuous outputs. In Proc. of ICLR.
Lee et al. (2017) Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
, pages 121–126, Copenhagen, Denmark. Association for Computational Linguistics.
Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81.
Lopez and Resnik (2005) Adam Lopez and Philip Resnik. 2005. Pattern visualization for machine translation output. In Proceedings of HLT/EMNLP 2005 Interactive Demonstrations, pages 12–13.
Madnani (2011) Nitin Madnani. 2011. ibleu: Interactively debugging and scoring statistical machine translation systems. In 2011 IEEE Fifth International Conference on Semantic Computing, pages 213–214. IEEE.
Mani (1999) Inderjeet Mani. 1999. Advances in automatic text summarization. MIT press.
Michel and Neubig (2018) Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium.
Mitchell et al. (2012) Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daume III. 2012. Midge: Generating image descriptions from computer vision detections. In Proc. EACL, pages 747–756.
Mohammad et al. (2016) Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. Journal of Artificial Intelligence Research
, 55:95–130.
Müller et al. (2018) Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proc. WMT, pages 61–72, Belgium, Brussels. Association for Computational Linguistics.
Murata et al. (2005) Masaki Murata, Kiyotaka Uchimoto, Qing Ma, Toshiyuki Kanamaru, and Hitoshi Isahara. 2005. Analysis of machine translation systems’ errors in tense, aspect, and modality. In Proc. PACLIC.
Naskar et al. (2011) Sudip Kumar Naskar, Antonio Toral, Federico Gaspari, and Andy Way. 2011. A framework for diagnostic evaluation of mt based on linguistic checkpoints. Proc. MT Summit, pages 529–536.
Neubig and Hu (2018) Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Proc. EMNLP, Brussels, Belgium.
Oh and Rudnicky (2000) Alice H Oh and Alexander I Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In Proceedings of the 2000 ANLP/NAACL Workshop on Conversational systems-Volume 3, pages 27–32. Association for Computational Linguistics.
Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, pages 311–318.
Popović (2015) Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Popović et al. (2006) Maja Popović, Adrià de Gispert, Deepa Gupta, Patrik Lambert, Hermann Ney, José B. Mariño, Marcello Federico, and Rafael Banchs. 2006. Morpho-syntactic information for automatic error analysis of statistical machine translation output. In Proc. WMT, pages 1–6.
Popović and Ney (2007) Maja Popović and Hermann Ney. 2007. Word error rates: Decomposition over POS classes and applications for error analysis. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 48–55.
Popović and Ney (2011) Maja Popović and Hermann Ney. 2011. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4):657–688.
Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proc. NAACL, New Orleans, USA.
Reiter and Dale (2000) Ehud Reiter and Robert Dale. 2000. Building natural language generation systems
. Cambridge university press.
Sachan and Neubig (2018) Devendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional translation models. In Proc. WMT, Brussels, Belgium.
Sennrich (2017) Rico Sennrich. 2017. How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs. In Proc. EACL, pages 376–382, Valencia, Spain.
Stymne (2011) Sara Stymne. 2011. BLAST: A tool for error analysis of machine translation output. In Proceedings of the ACL-HLT 2011 System Demonstrations, pages 56–61.
Vilar et al. (2006) David Vilar, Jia Xu, Luis Fernando d’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proc. LREC, pages 697–702.
Wang et al. (2018) Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. 2018. A tree-based decoder for neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium.
Weese and Callison-Burch (2010) Jonathan Weese and Chris Callison-Burch. 2010. Visualizing data structures in parsing-based machine translation. The Prague Bulletin of Mathematical Linguistics, 93:127–136.
Zeman et al. (2011) Daniel Zeman, Mark Fishel, Jan Berka, and Ondřej Bojar. 2011. Addicter: What is wrong with my translations? The Prague Bulletin of Mathematical Linguistics, 96(1):79–88.
Zhou et al. (2008) Ming Zhou, Bo Wang, Shujie Liu, Mu Li, Dongdong Zhang, and Tiejun Zhao. 2008. Diagnostic evaluation of machine translation systems using automatically constructed linguistic check-points. In Proc. COLING, pages 1121–1128, Manchester, UK. Coling 2008 Organizing Committee.

Appendix A Example Command

Fig. 7 shows an example of the command that was used to generate the report containing the figures and tables used in this paper.

compare-mt: A Tool for Holistic Comparison of Language Generation Systems

Variance-Aware Machine Translation Test Sets

TypeShift: A User Interface for Visualizing the Typing Production Process

Difficulty-Aware Machine Translation Evaluation

Noisy-parallel and comparable corpora filtering methodology for the extraction of bi-lingual equivalent data at sentence level

A Sentence Meaning Based Alignment Method for Parallel Text Corpora Preparation

Fast Nearest Neighbor Machine Translation

Kafka versus RabbitMQ

Code Repositories

compare-mt