Sun, Z;
Zhang, JM;
Harman, M;
Papadakis, M;
Zhang, L;
(2020)
Automatic Testing and Improvement of Machine Translation.
In:
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering.
(pp. pp. 974-985).
ACM: Seoul, South Korea.
Preview |
Text
NMT_Testing_rts.pdf - Accepted Version Download (813kB) | Preview |
Abstract
This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic testing to detect inconsistency bugs (without access to human oracles). It then adopts probability-reference or cross-reference to post-process the translations, in a grey-box or black-box manner, to repair the inconsistencies. Our evaluation on two state-of-the-art translators, Google Translate and Transformer, indicates that TransRepair has a high precision (99%) on generating input pairs with consistent translations. With these tests, using automatic consistency metrics and manual assessment, we find that Google Translate and Transformer have approximately 36% and 40% inconsistency bugs. Black-box repair fixes 28% and 19% bugs on average for Google Translate and Transformer. Grey-box repair fixes 30% bugs on average for Transformer. Manual inspection indicates that the translations repaired by our approach improve consistency in 87% of cases (degrading it in 2%), and that our repairs have better translation acceptability in 27% of the cases (worse in 8%).
| Type: | Proceedings paper |
|---|---|
| Title: | Automatic Testing and Improvement of Machine Translation |
| Event: | International Conference on Software Engineering 2020 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3377811.3380420 |
| Publisher version: | https://doi.org/10.1145/3377811.3380420 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10099086 |
Archive Staff Only
 |
View Item |
