Investigating the contribution of linguistic information to quality estimation
- PDF / 696,554 Bytes
- 20 Pages / 439.37 x 666.142 pts Page_size
- 18 Downloads / 207 Views
Investigating the contribution of linguistic information to quality estimation Mariano Felice · Lucia Specia
Received: 14 October 2012 / Accepted: 19 April 2013 / Published online: 30 August 2013 © Springer Science+Business Media Dordrecht 2013
Abstract This paper describes a study on the contribution of linguistically-informed features to the task of quality estimation for machine translation at sentence level. A standard regression algorithm is used to build models using a combination of linguistic and non-linguistic features extracted from the input text and its machine translation. Experiments with three English–Spanish translation datasets show that linguistic features on their own are not able to outperform shallower features based on statistics from the input text, its translation and additional corpora. However, further analysis suggests that linguistic information can be useful to produce better results if carefully combined with other features. An in-depth analysis of the results highlights a number of issues related to the use of linguistic features. Keywords
Machine translation · Evaluation · Quality estimation
1 Introduction Estimating the quality of automatic translations has become a subject of increasing interest within the machine translation (MT) community for a number of reasons, such as helping human translators post-editing MT, warning users about non-reliable translations or combining output from multiple MT systems. Different from most classic approaches for measuring the progress of an MT system or comparing MT systems
M. Felice (B) Computer Laboratory, University of Cambridge, 15 JJ Thomson Avenue, Cambridge CB3 0FD, UK e-mail: [email protected]; [email protected] L. Specia Department of Computer Science, University of Sheffield, Regent Court, 211 Portobello, Sheffield S1 4DP, UK e-mail: [email protected]
123
194
M. Felice, L. Specia
such as BLEU (Papineni et al. 2002), which assess quality by contrasting system output to reference translations, quality estimation (QE) is a more challenging task, aimed at MT systems in use, and therefore without access to reference translations. From the findings of previous work on reference-dependent MT evaluation, it is clear that metrics exploiting linguistic information can achieve significantly better correlation with human judgements on quality, particularly at the level of sentences (Giménez and Màrquez 2010). Intuitively, this should also apply for QE metrics. However, while evaluation metrics compare linguistic representations of the system output and reference translations in a single language (e.g. matching of n-grams of part-ofspeech (PoS) tags or predicate–argument structures), QE metrics need to perform a more complex cross-lingual comparison over linguistic representations of the source and translation texts. The hypothesis put forward in this paper is that using linguistic information to contrast the source and translation texts can be beneficial for QE. We investigate how linguistic information affects transl
Data Loading...
