الفهرس 
Machine Translation: History, Challenges and EvaluationOnly 14 pages are availabe for public view
 |  | from | 176 |  |  |
| | | from | 176 | | |
Abstract
Summary
Over the last decade, artificial intelligence applications have revolutionized many industries and natural language processing is no exception. Machine translation in particular witnessed great leaps in quality. Language barriers are no longer insurmountable. People are one keystroke away from translating any texts written in languages they do not speak. They might not get high-quality translation; but it is undoubtedly a step forward towards deciphering a text they would have never understood if it were not for machine translation.
Machine translation has been notorious for bad quality over the larger part of its history. However, the use of neural networks in machine translation yielded major quality gains, especially when compared to the previous predominant paradigm of statistical machine translation. Many studies reported improved performance for many language pairs, especially when translating between European languages. English-to-Arabic translation, however, remains one of the least studied language directions in the field.
Consequently, the present study performs a quality assessment of two machine translation systems: neural machine translation (NMT) and statistical machine translation (SMT). It aims to highlight the strengths and weaknesses of both systems, and the patterns which trigger errors in their outputs. It also sheds light on the gains NMT brings in terms of increasing productivity or reducing the costs of post-editing compared to translating from scratch.
The translation quality assessment is carried out on two levels: the first is a linguistic error analysis which quantifies the errors produced by each system, their types, and severity. It then examines the linguistic patterns that trick the system into producing an erroneous translation. The analysis is based on the DQF-MQM Harmonized Error Typology, a joint effort by the Translation Automation User Society (TAUS) and the German Research Center for Artificial Intelligence (DFKI). The second level of assessment measures the temporal, technical, and cognitive post-editing efforts, as defined by Hans Peter Krings in Repairing Texts (2001).
The corpus of analysis consists of six articles extracted from WikiHow, a website that provides step-by-step guides on a variety of topics. Each article is co-authored, edited, and revised by around 40 persons, according to the website. The selected articles cover three domains: cryptocurrency, cybersecurity, and healthcare. They feature a simple precise language with a plethora of technical terms. These articles are then machine-translated into Arabic twice: once with Google’s NMT and another with Google’s SMT.
The thesis consists of an introduction, four chapters, and a conclusion. The introduction presents the research problems, motivations, objectives, and questions. It also reviews relevant research literature in the field of machine translation quality assessment, especially those focusing on Arabic. The literature review concludes that most studies report improved quality when using NMT, most studies conducted on Arabic focus on translation from Arabic and not vice versa, research comparing NMT to SMT for Arabic is very limited, and finally no studies have conducted a fine-grained error analysis and evaluation of the post-editing effort as the one performed in this study to compare the quality of NMT to that of SMT for Arabic. The introduction finally provides a detailed presentation of the research methodology, procedures, and tools used in the analysis.
Chapter One starts with an overview of machine translation history and the general concepts underlying the two systems under analysis. It also discusses the problems of automating translation from three angles: data, language, and Arabic language in particular. It finally reviews automatic and human methods of evaluating machine translation quality, their advantages and disadvantages, and the most used assessment tools.
Chapters Two and Three provide the linguistic analysis which classifies the errors produced by each system in terms of quantity, severity, types, and triggers. The errors are arranged in a descending order from the most to least common, supported by examples from the corpus and followed by an explanation of the linguistic or technical phenomenon behind the error.
Chapter Four presents a comparative analysis of the post-editing effort required for each system in terms of the cost, time spent, the amount and type of edits made, and the number keys pressed during the editing process. Such analysis shows whether using NMT helps reduce any of these factors, therefore, increasing productivity and reducing costs.
Finally, the conclusion answers the research questions based on the results driven from the analysis chapters. It confirms that using NMT for English-into-Arabic translation has indeed improved quality compared to SMT. The conclusion also highlights the particular aspects where NMT has exceptionally outperformed SMT in addition to those which remain challenging for both systems. Moreover, it discusses the benefits language service providers will reap from using NMT. The conclusion also stresses the importance of dispelling the common misconception about machine translation and its quality, and the paramount need of teaching post-editing as a discipline to translation students. The best defense against fraud is education. Including post-editing in translator training programs will help them make optimal use of machine translation while protecting them from falling victim to underpayment.