Publications

2025

1. Metronome: tracing variation in poetic meters via local sequence alignment

   Ben Nagy, Artjoms Šeļa, Mirella De Sisto, and Petr Plecháč

   Computational Humanities Research, vol. 1, pp. e1, 2025

   [DOI]
2. An Interdisciplinary Approach to Human-Centered Machine Translation

   Marine Carpuat, Omri Asscher, Kalika Bali, Luisa Bentivogli, Frédéric Blain, Lynne Bowker, Monojit Choudhury, Hal Daumé III, Kevin Duh, Ge Gao, Alvin II Grissom, Marzena Karpinska, Elaine C. Khoong, William D. Lewis, André F.T. Martins, Mary Nurminen, Douglas W. Oard, Maja Popovic, Michel Simard, and François Yvon

   arXiv preprint arXiv:2506.13468, 2025

   [Abs] [URL]

   Machine Translation (MT) tools are widely used today, often in contexts where professional translators are not present. Despite progress in MT technology, a gap persists between system development and real-world usage, particularly for non-expert users who may struggle to assess translation reliability. This paper advocates for a human-centered approach to MT, emphasizing the alignment of system design with diverse communicative goals and contexts of use. We survey the literature in Translation Studies and Human-Computer Interaction to recontextualize MT evaluation and design to address the diverse real-world scenarios in which MT is used today.

3. Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation

   Chiara Manna, Afra Alishahi, Frédéric Blain, and Eva Vanmassenhove

   In Proceedings of the 3rd Workshop on Gender-Inclusive Translation Technologies (GITT 2025), pp. 1–16, Jun, 2025

   [Abs] [URL]

   While gender bias in modern Neural Machine Translation (NMT) systems has received much attention, the traditional evaluation metrics for these systems do not fully capture the extent to which models integrate contextual gender cues. We propose a novel evaluation metric called Minimal Pair Accuracy (MPA) which measures the reliance of models on gender cues for gender disambiguation. We evaluate a number of NMT models using this metric, we show that they ignore available gender cues in most cases in favour of (statistical) stereotypical gender interpretation. We further show that in anti-stereotypical cases, these models tend to more consistently take male gender cues into account while ignoring the female cues. Finally, we analyze the attention head weights in the encoder component of these models and show that while all models to some extent encode gender information, the male gender cues elicit a more diffused response compared to the more concentrated and specialized responses to female gender cues.

4. Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help

   Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Dattatray Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch

   In Proceedings of the Tenth Conference on Machine Translation (WMT 2025), pp. 414–461, Nov, 2025

   [Abs] [URL]

   The WMT25 Shared Task on Automated Translation Evaluation Systems evaluates metrics and quality estimation systems that assess the quality of language translation systems. This task unifies and consolidates the separate WMT shared tasks on Machine Translation Evaluation Metrics and Quality Estimation from previous years. Our primary goal is to encourage the development and assessment of new state-of-the-art translation quality evaluation systems. The shared task this year consisted of three subtasks: (1) segment-level quality score prediction, (2) span-level translation error annotation, and (3) quality-informed segment-level error correction. The evaluation data for the shared task were provided by the General MT shared task and were complemented by "challenge sets" from both the organizers and participants. Task 1 results indicate the strong performance of large LLMs at the system level, while reference-based baseline metrics outperform LLMs at the segment level. Task 2 results indicate that accurate error detection and balancing precision and recall are persistent challenges. Task 3 results show that minimal editing is challenging even when informed by quality indicators. Robustness across the broad diversity of languages remains a major challenge across all three subtasks.

2024

1. XSL-HoReCo and GoSt-ParC-Sign: Two New Signed Language - Written Language Parallel Corpora

   Mirella De Sisto, Vincent Vandeghinste, Caro Brosens, Myriam Vermeerbergen, and Dimitar Shterionov

   Selected papers from the CLARIN Annual Conference 2023, 2024

   [Abs] [URL] [DOI]

   Developments in language technology targeting signed languages are lagging behind in comparison to the advances related to what is available for so-called spoken languages. This is partly due to the scarcity of good quality signed language data, including good quality parallel corpora of signed and spoken languages. This paper introduces two parallel corpora which aim at reducing the gap between signed and spoken-only language technology: The XSL Hotel Review Corpus (XSL-HoReCo) and the Gold Standard Parallel Corpus of Signed and Spoken Language (GoSt-ParC-Sign). Both corpora are available through the CLARIN infrastructure.

2. PoeTree: Poetry Treebanks in Czech, English, French, German, Hungarian, Italian, Portuguese, Russian, Slovenian and Spanish

   Petr Plecháč, Silvie Cinková, Robert Kolár, Artjoms Šeļa, Mirella De Sisto, Lara Nugues, Thomas Haider, and Neža Kočnik

   Research Data Journal for the Humanities and Social Sciences, pp. 1 - 17, 2024

   [Abs] [URL] [DOI]

   This article presents a set of standardised corpora of poetry comprising over 330,000 poems in ten languages (Czech, English, French, German, Hungarian, Italian, Portuguese, Russian, Slovenian, and Spanish). Each corpus has been deduplicated, enriched with Universal Dependencies, provided with additional metadata, and converted into a unified json structure.

3. What do Large Language Models Need for Machine Translation Evaluation?

   Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orăsan, Tharindu Ranasinghe, and Frédéric Blain

   2024

   [Abs] [URL]

   Leveraging large language models (LLMs) for various natural language processing tasks has led to superlative claims about their performance. For the evaluation of machine translation (MT), existing research shows that LLMs are able to achieve results comparable to fine-tuned multilingual pre-trained language models. In this paper, we explore what translation information, such as the source, reference, translation errors and annotation guidelines, is needed for LLMs to evaluate MT quality. In addition, we investigate prompting techniques such as zero-shot, Chain of Thought (CoT) and few-shot prompting for eight language pairs covering high-, medium- and low-resource languages, leveraging varying LLM variants. Our findings indicate the importance of reference translations for an LLM-based evaluation. While larger models do not necessarily fare better, they tend to benefit more from CoT prompting, than smaller models. We also observe that LLMs do not always provide a numerical score when generating evaluations, which poses a question on their reliability for the task. Our work presents a comprehensive analysis for resource-constrained and training-less LLM-based evaluation of machine translation. We release the accrued prompt templates, code and data publicly for reproducibility.

4. DORE: A Dataset For Portuguese Definition Generation

   Anna Beatriz Dimas Furtado, Tharindu Ranasinghe, Frédéric Blain, and Ruslan Mitkov

   2024

   [Abs] [URL]

   Definition modelling (DM) is the task of automatically generating a dictionary definition for a specific word. Computational systems that are capable of DM can have numerous applications benefiting a wide range of audiences. As DM is considered a supervised natural language generation problem, these systems require large annotated datasets to train the machine learning (ML) models. Several DM datasets have been released for English and other high-resource languages. While Portuguese is considered a mid/high-resource language in most natural language processing tasks and is spoken by more than 200 million native speakers, there is no DM dataset available for Portuguese. In this research, we fill this gap by introducing DORE; the first dataset for Definition MOdelling for PoRtuguEse containing more than 100,000 definitions. We also evaluate several deep learning based DM models on DORE and report the results. The dataset and the findings of this paper will facilitate research and study of Portuguese in wider contexts.

5. Understanding poetry using natural language processing tools: a survey

   Mirella De Sisto, Laura Hernández-Lorenzo, Javier De la Rosa, Salvador Ros, and Elena González-Blanco

   Digital Scholarship in the Humanities, pp. fqae001, Feb, 2024

   [Abs] [URL] [DOI]

   Analyzing poetry with automatic tools has great potential for improving verse-related research. Over the last few decades, this field has expanded notably and a large number of tools aiming at analyzing various aspects of poetry have been developed. However, the concrete connection between these tools and traditional scholars investigating poetry and metrics is often missing. The purpose of this article is to bridge this gap by providing a comprehensive survey of the automatic poetry analysis tools available for European languages. The tools are described and classified according to the language for which they are primarily developed, and to their functionalities and purpose. Particular attention is given to those that have open-source code or provide an online version with the same functionality. Combining more traditional research with these tools has clear advantages: it provides the opportunity to address theoretical questions with the support of large amounts of data; also, it allows for the development of new and diversified approaches.

6. Microvariation in the second form of the infinitive in Campania: the case of Valle Caudina

   Kim Groothuis, and Mirella De Sisto

   Isogloss, vol. 10, Mar, 2024

   [Abs] [URL] [DOI]

   Campanian dialects such as Neapolitan feature a so-called ‘second form of the infinitive’ (SFI), a form consisting of the bare verbal stem, which can be used after functional verbs. This paper addresses the microvariation concerning the construction by analysing novel data from the Valle Caudina, located to the northeast of Naples. The SFI is frequently found specifically with the imperative va ‘go!’. In Neapolitan, the form has been reanalysed as an imperatival form in this context, yielding an asyndetic imperative. At a first glance, the use of the SFI in Valle Caudina looks very similar to its Neapolitan counterpart, but unlike Neapolitan, the SFI in these varieties has remained non-finite and has not been reanalysed as an imperative. These dialects can thus be considered a previous stage of the development described for Neapolitan by Ledgeway (1997, 2007, 2009). This claim finds support in the absence of metaphonetic forms - which have appeared in Neapolitan, as a consequence of the reanalysis - as well as the presence of clitic climbing. Finally, unlike Neapolitan, the SFI is becoming less productive in the varieties of the Valle Caudina.

7. Forged-GAN-BERT: Authorship Attribution for LLM-Generated Forged Novels

   Kanishka Silva, Ingo Frommholz, Burcu Can, Frédéric Blain, Raheem Sarwar, and Laura Ugolini

   In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 325–337, Mar, 2024

   [Abs] [URL] [DOI]

   The advancement of generative Large Language Models (LLMs), capable of producing human-like texts, introduces challenges related to the authenticity of the text documents. This requires exploring potential forgery scenarios within the context of authorship attribution, especially in the literary domain. Particularly,two aspects of doubted authorship may arise in novels, as a novel may be imposed by a renowned author or include a copied writing style of a well-known novel. To address these concerns, we introduce Forged-GAN-BERT, a modified GANBERT-based model to improve the classification of forged novels in two data-augmentation aspects: via the Forged Novels Generator (i.e., ChatGPT) and the generator in GAN. Compared to other transformer-based models, the proposed Forged-GAN-BERT model demonstrates an improved performance with F1 scores of 0.97 and 0.71 for identifying forged novels in single-author and multi-author classification settings. Additionally, we explore different prompt categories for generating the forged novels to analyse the quality of the generated texts using different similarity distance measures, including ROUGE-1, Jaccard Similarity, Overlap Confident, and Cosine Similarity.

8. Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation

   Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck

   In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pp. 88–101, Sep, 2024

   [Abs] [URL]

   The quality of output from large language models (LLMs), particularly in machine translation (MT), is closely tied to the quality of in-context examples (ICEs) provided along with the query, i.e., the text to translate. The effectiveness of these ICEs is influenced by various factors, such as the domain of the source text, the order in which the ICEs are presented, the number of these examples, and the prompt templates used. Naturally, selecting the most impactful ICEs depends on understanding how these affect the resulting translation quality, which ultimately relies on translation references or human judgment. This paper presents a novel methodology for in-context learning (ICL) that relies on a search algorithm guided by domain-specific quality estimation (QE). Leveraging the XGLM model, our methodology estimates the resulting translation quality without the need for translation references, selecting effective ICEs for MT to maximize translation quality. Our results demonstrate significant improvements over existing ICL methods and higher translation performance compared to fine-tuning a pre-trained language model (PLM), specifically mBART-50.

9. Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task

   Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie

   In Proceedings of the Ninth Conference on Machine Translation, pp. 47–81, Nov, 2024

   [Abs] [URL] [DOI]

   The WMT24 Metrics Shared Task evaluated the performance of automatic metrics for machine translation (MT), with a major focus on LLM-based translations that were generated as part of the WMT24 General MT Shared Task. As LLMs become increasingly popular in MT, it is crucial to determine whether existing evaluation metrics can accurately assess the output of these systems.To provide a robust benchmark for this evaluation, human assessments were collected using Multidimensional Quality Metrics (MQM), continuing the practice from recent years. Furthermore, building on the success of the previous year, a challenge set subtask was included, requiring participants to design contrastive test suites that specifically target a metric’s ability to identify and penalize different types of translation errors.Finally, the meta-evaluation procedure was refined to better reflect real-world usage of MT metrics, focusing on pairwise accuracy at both the system- and segment-levels.We present an extensive analysis on how well metrics perform on three language pairs: English to Spanish (Latin America), Japanese to Chinese, and English to German. The results strongly confirm the results reported last year, that fine-tuned neural metrics continue to perform well, even when used to evaluate LLM-based translation systems.

10. Findings of the Quality Estimation Shared Task at WMT 2024: Are LLMs Closing the Gap in QE?

    Chrysoula Zerva, Frédéric Blain, José G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guerreiro, Giuseppe Attanasio, Ricardo Rei, Constantin Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, and André Martins

    In Proceedings of the Ninth Conference on Machine Translation, pp. 82–109, Nov, 2024

    [Abs] [URL] [DOI]

    We report the results of the WMT 2024 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. In this edition, we expanded our scope to assess the potential for quality estimates to help in the correction of translated outputs, hence including an automated post-editing (APE) direction. We publish new test sets with human annotations that target two directions: providing new Multidimensional Quality Metrics (MQM) annotations for three multi-domain language pairs (English to German, Spanish and Hindi) and extending the annotations on Indic languages providing direct assessments and post edits for translation from English into Hindi, Gujarati, Tamil and Telugu. We also perform a detailed analysis of the behaviour of different models with respect to different phenomena including gender bias, idiomatic language, and numerical and entity perturbations. We received submissions based both on traditional, encoder-based approaches as well as large language model (LLM) based ones.

2023

1. Evaluating the Effectiveness of Pre-trained Language Models in Predicting the Helpfulness of Online Product Reviews

   Ali Boluki, Javad Pourmostafa Roshan Sharami, and Dimitar Shterionov

   2023

   [Abs] [URL]

   Businesses and customers can gain valuable information from product reviews. The sheer number of reviews often necessitates ranking them based on their potential helpfulness. However, only a few reviews ever receive any helpfulness votes on online marketplaces. Sorting all reviews based on the few existing votes can cause helpful reviews to go unnoticed because of the limited attention span of readers. The problem of review helpfulness prediction is even more important for higher review volumes, and newly written reviews or launched products. In this work we compare the use of RoBERTa and XLM-R language models to predict the helpfulness of online product reviews. The contributions of our work in relation to literature include extensively investigating the efficacy of state-of-the-art language models – both monolingual and multilingual – against a robust baseline, taking ranking metrics into account when assessing these approaches, and assessing multilingual models for the first time. We employ the Amazon review dataset for our experiments. According to our study on several product categories, multilingual and monolingual pre-trained language models outperform the baseline that utilizes random forest with handcrafted features as much as 23% in RMSE. Pre-trained language models reduce the need for complex text feature engineering. However, our results suggest that pre-trained multilingual models may not be used for fine-tuning only one language. We assess the performance of language models with and without additional features. Our results show that including additional features like product rating by the reviewer can further help the predictive methods.

2. SignON: Sign Language Translation. Progress and challenges

   Vincent Vandeghinste, Dimitar Shterionov, Mirella De Sisto, Aoife Brady, Mathieu De Coster, Lorraine Leeson, Josep Blat, Frankie Picron, Marcello Paolo Scipioni, Aditya Parikh, Louis Bosch, John O’Flaherty, Joni Dambre, Jorn Rijckaert, Bram Vanroy, Victor Ubieto Nogales, Santiago Egea Gomez, Ineke Schuurman, Gorka Labaka, Adrian Nunez-Marcos, Irene Murtagh, Euan McGill, and Horacio Saggion

   In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 2023

   [URL]
3. GoSt-ParC-Sign: Gold Standard Parallel Corpus of Sign and spoken language

   Mirella De Sisto, Vincent Vandeghinste, Lien Soetemans, Caro Brosens, and Dimitar Shterionov

   In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 503–504, 2023

   [URL]
4. Report on Europe’s Sign Languages. Deliverable European Language Equality II project

   Vincent Vandeghinste, Mirella De Sisto, Maria Kopf, Marc Schulder, Caro Brosens, Lien Soetemans, Rehana Omardeen, Frankie Picron, Davy Van Landuyt, Irene Murtagh, Eleftherios Avramidis, and Mathieu De Coster

   2023

   [URL]
5. NGT-HoReCo and GoSt-ParC-Sign: Two new Sign Language - Spoken Language parallel corpora

   Mirella De Sisto, Vincent Vandeghinste, Dimitar Shterionov, Lien Soetemans, and Caro Brosens

   In CLARIN Annual Conference Proceedings, 2023, pp. 6–9, 2023

   [URL]
6. The development of a poetic tradition. A study on a Dutch Renaissance Poetry Corpus

   Mirella De Sisto

   Studia Metrica et Poetica, vol. 10, pp. 36–68, 2023

   [URL] [DOI]
7. Tailoring Domain Adaptation for Machine Translation Quality Estimation

   Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, Frédéric Blain, Eva Vanmassenhove, Mirella De Sisto, Chris Emmery, and Pieter Spronck

   In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 9–20, Jun, 2023

   [Abs] [URL]

   While quality estimation (QE) can play an important role in the translation process, its effectiveness relies on the availability and quality of training data. For QE in particular, high-quality labeled data is often lacking due to the high-cost and effort associated with labeling such data. Aside from the data scarcity challenge, QE models should also be generalizabile, i.e., they should be able to handle data from different domains, both generic and specific. To alleviate these two main issues — data scarcity and domain mismatch — this paper combines domain adaptation and data augmentation within a robust QE system. Our method is to first train a generic QE model and then fine-tune it on a specific domain while retaining generic knowledge. Our results show a significant improvement for all the language pairs investigated, better cross-lingual inference, and a superior performance in zero-shot learning scenarios as compared to state-of-the-art baselines.

8. A New English-Dutch-NGT Corpus for the Hospitality Domain

   Mirella De Sisto, Vincent Vandeghinste, and Dimitar Shterionov

   In Proceedings of the Second International Workshop on Automatic Translation for Signed and Spoken Languages, pp. 34–37, Jun, 2023

   [Abs] [URL]

   One of the major challenges hampering the development of language technology which targets sign languages is the extremely limited availability of good quality data geared towards machine learning and deep learning approaches. In this paper we introduce the NGT-Dutch Hotel Review Corpus (NGT-HoReCo), which addresses this issue by providing multimodal parallel data in English, Dutch and Sign Language of the Netherlands (NGT). The corpus contains 283 hotel reviews in written English, translated into written Dutch and into NGT videos. It will be made publicly available through CLARIN and through the ELG platform.

9. Authorship Attribution of Late 19th Century Novels using GAN-BERT

   Kanishka Silva, Burcu Can, Frédéric Blain, Raheem Sarwar, Laura Ugolini, and Ruslan Mitkov

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 310–320, Jul, 2023

   [Abs] [URL] [DOI]

   Authorship attribution aims to identify the author of an anonymous text. The task becomes even more worthwhile when it comes to literary works. For example, pen names were commonly used by female authors in the 19th century resulting in some literary works being incorrectly attributed or claimed. With this motivation, we collated a dataset of late 19th century novels in English. Due to the imbalance in the dataset and the unavailability of enough data per author, we employed the GANBERT model along with data sampling strategies to fine-tune a transformer-based model for authorship attribution. Differently from the earlier studies on the GAN-BERT model, we conducted transfer learning on comparatively smaller author subsets to train more focused author-specific models yielding performance over 0.88 accuracy and F1 scores. Furthermore, we observed that increasing the sample size has a negative impact on the model’s performance. Our research mainly contributes to the ongoing authorship attribution research using GAN-BERT architecture, especially in attributing disputed novelists in the late 19th century.

10. Text Data Augmentation Using Generative Adversarial Networks – A Systematic Review

    Kanishka Silva, Burcu Can, Raheem Sarwar, Frédéric Blain, and Ruslan Mitkov

    Journal of Computational and Applied Linguistics, vol. 1, pp. 6–38, Jul, 2023

    [Abs] [DOI]

    Insufficient data is one of the main drawbacks in natural language processing tasks, and the most prevalent solution is to collect a decent amount of data that will be enough for the optimisation of the model. However, recent research directions are strategically moving towards increasing training examples due to the nature of the data-hungry neural models. Data augmentation is an emerging area that aims to ensure the diversity of data without attempting to collect new data exclusively to boost a model’s performance. Limitations in data augmentation, especially for textual data, are mainly due to the nature of language data, which is precisely discrete. Generative Adversarial Networks (GANs) were initially introduced for computer vision applications, aiming to generate highly realistic images by learning the image representations. Recent research has focused on using GANs for text generation and augmentation. This systematic review aims to present the theoretical background of GANs and their use for text augmentation alongside a systematic review of recent textual data augmentation applications such as sentiment analysis, low resource language generation, hate speech detection and fraud review analysis. Further, a notion of challenges in current research and future directions of GAN-based text augmentation are discussed in this paper to pave the way for researchers especially working on low-text resources.

11. Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent

    Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster

    In Proceedings of the Eighth Conference on Machine Translation, pp. 578–628, Dec, 2023

    [Abs] [URL] [DOI]

    This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year’s success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks. We present an extensive analysis on how well metrics perform on three language pairs: Chinese-English, Hebrew-English on the sentence-level and English-German on the paragraph-level. The results strongly confirm the results reported last year, that neural-based metrics are significantly better than non-neural metrics in their levels of correlation with human judgments. Further, we investigate the impact of bad reference translations on the correlations of metrics with human judgment. We present a novel approach for generating synthetic reference translations based on the collection of MT system outputs and their corresponding MQM ratings, which has the potential to mitigate bad reference issues we observed this year for some language pairs. Finally, we also study the connections between the magnitude of metric differences and their expected significance in human evaluation, which should help the community to better understand and adopt new metrics.

12. Findings of the WMT 2023 Shared Task on Quality Estimation

    Frédéric Blain, Chrysoula Zerva, Ricardo Ribeiro, Nuno M. Guerreiro, Diptesh Kanojia, José G. Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and André Martins

    In Proceedings of the Eighth Conference on Machine Translation, pp. 629–653, Dec, 2023

    [Abs] [URL] [DOI]

    We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test-set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions.

13. Quality Estimation-Assisted Automatic Post-Editing

    Sourabh Deoghare, Diptesh Kanojia, Frédéric Blain, Tharindu Ranasinghe, and Pushpak Bhattacharyya

    In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1686–1698, Dec, 2023

    [Abs] [URL] [DOI]

    Automatic Post-Editing (APE) systems are prone to over-correction of the Machine Translation (MT) outputs. While Word-level Quality Estimation (QE) system can provide a way to curtail the over-correction, a significant performance gain has not been observed thus far by utilizing existing APE and QE combination strategies. In this paper, we propose joint training of a model on APE and QE tasks to improve the APE. Our proposed approach utilizes a multi-task learning (MTL) methodology, which shows significant improvement while treating both tasks as a ‘bargaining game’ during training. Moreover, we investigate various existing combination strategies and show that our approach achieves state-of-the-art performance for a ‘distant’ language pair, viz., English-Marathi. We observe an improvement of 1.09 TER and 1.37 BLEU points over a baseline QE-Unassisted APE system for English-Marathi, while also observing 0.46 TER and 0.62 BLEU points for English-German. Further, we discuss the results qualitatively and show how our approach helps reduce over-correction, thereby improving the APE performance. We also observe that the degree of integration between QE and APE directly correlates with the APE performance gain. We release our code and models publicly.

2022

1. The Ecological Footprint of Neural Machine Translation Systems

   Dimitar Shterionov, and Eva Vanmassenhove

   2022

   [URL] [DOI]
2. Modelli di demarcazione della metà verso nel metro rinascimentale romanzo

   Mirella De Sisto

   ILLA - Nuove Ricerche Umanistiche, vol. Interruzioni e cesure. Fenomeni e pratiche della discontinuità in linguistica, letteratura e arti performative, pp. 192–200, 2022

   [Abs] [URL]

   The present paper discusses two types of mid-line marking attested in Romance traditions instantiations of Renaissance metre and outlines an explanation for their structural differences. On the one hand, some poetic traditions, namely, French and Catalan, preserved the caesura of the source form. On the other hand, Italian, Spanish and Portuguese poetry developed a kind of mid-line marking which is based on the obligatory prominence of a metrical position in the middle of the line. A further distinction can be made when considering that French and Catalan caesurae are not exactly the same: in French, this plays the role of the only metrical element smaller than the line; ln Catalan, instead, a prominent position line-medially is also dividing the line in two hemistiches. Catalan caesura constitutes a hybrid form between a proper caesura, in the French way, and a prominence mid-line marking system.

3. Quality Estimation for the Translation Industry – Data Challenges

   Javad Pourmostafa Roshan Sharami, Elena Murgolo, and Dimitar Shterionov

   In The 32nd Meeting of Computational Linguistics in The Netherlands, CLIN, Jun, 2022

   [Abs] [URL] [DOI]

   Machine Translation (MT) has become an irreplaceable part of translation industry workflows. With a direct impact on productivity, it is very important for human post-editors and project managers to be informed about the translation quality of MT.MT Quality estimation (QE) is the task of predicting the quality of a translation without human references. For the translation industry QE can be used as an indicator of the amount of post-editing needed as well as of productivity. QE can be applied at word-, sentence- or document-level. In the cases of sentence- and document-level QE, given a source text and its MT counterpart the task is to predict a score (typically TER) that indicates the translation quality. As with most NLP tasks nowadays, state-of-the-art QE has been achieved using DL methods. Optimal QE performance is not only a question of model architecture and hyperparameters but it strongly depends on the quantity and quality of the data.In a business environment, such as the one of the translation industry models or system that are used in production should adhere to economic and usability criteria. QE for the translation industry should be optimised for domain- and use-case specific data, should be efficient and be adaptable. In a collaborative project between Orbital14 and Tilburg University, funded by Aglatech14, we develop a framework for MT quality assessment (MTQA) which strongly relies on QE. In this project we focus on a specific domain (patents, IP) and a language pair (English-Italian).In this work we present our approach to data collection, analysis and preprocessing prior to building QE models. We started with proprietary data provided by Aglatech14 to identify specific patterns that need to be covered by the QE tool. As the volume of initial data was not sufficient to build robust DL models (42K source-translation-post-edit triplets, from which we compute the TER scores) we added a corpus of 127K source-human translation sentences. We used Aglatech14’s translation model to translate the source and generate a pseudo corpus of triplets (source-MT-post-edit), which were then used to compute TER scores. To further extend our data set we used publicly available data – a generic English- Italian corpus containing  105M sentence pairs. Aiming at a domain-specific QE, we select only similar to the industry data sentence pairs. We used a ranking data selection method based on cosine similarity to selected additional 42K sentences – those with the highest similarity score to the Aglatech14 data. After data selection, wetranslated the selected source data to create new synthetic data comprising of source, machine translation and target sentences. We consider the target sentences as post-edited sentences. As in the previous case, we computed the TER score between the MT and the target to use as labels for training our QE models.We use these data to build state-of-the-art QE models and evaluate their performance against a gold standard reference set, provided by Aglatech14.

4. Challenges with Sign Language Datasets for Sign Language Recognition and Translation

   Mirella De Sisto, Vincent Vandeghinste, Santiago Egea Gómez, Mathieu De Coster, Dimitar Shterionov, and Horacio Saggion

   In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2478–2487, Jun, 2022

   [Abs] [URL]

   Sign Languages (SLs) are the primary means of communication for at least half a million people in Europe alone. However, the development of SL recognition and translation tools is slowed down by a series of obstacles concerning resource scarcity and standardization issues in the available data. The former challenge relates to the volume of data available for machine learning as well as the time required to collect and process new data. The latter obstacle is linked to the variety of the data, i.e., annotation formats are not unified and vary amongst different resources. The available data formats are often not suitable for machine learning, obstructing the provision of automatic tools based on neural models. In the present paper, we give an overview of these challenges by comparing various SL corpora and SL machine learning datasets. Furthermore, we propose a framework to address the lack of standardization at format level, unify the available resources and facilitate SL research for different languages. Our framework takes ELAN files as inputs and returns textual and visual data ready to train SL recognition and translation models. We present a proof of concept, training neural translation models on the data produced by the proposed framework.

5. A Quality Estimation and Quality Evaluation Tool for the Translation Industry

   Elena Murgolo, Javad Pourmostafa Roshan Sharami, and Dimitar Shterionov

   In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pp. 307–308, Jun, 2022

   [Abs] [URL]

   With the increase in machine translation (MT) quality over the latest years, it has now become a common practice to integrate MT in the workflow of language service providers (LSPs) and other actors in the translation industry. With MT having a direct impact on the translation workflow, it is important not only to use high-quality MT systems, but also to understand the quality dimension so that the humans involved in the translation workflow can make informed decisions. The evaluation and monitoring of MT output quality has become one of the essential aspects of language technology management in LSPs’ workflows. First, a general practice is to carry out human tests to evaluate MT output quality before deployment. Second, a quality estimate of the translated text, thus after deployment, can inform post editors or even represent post-editing effort. In the former case, based on the quality assessment of a candidate engine, an informed decision can be made whether the engine would be deployed for production or not. In the latter, a quality estimate of the translation output can guide the human post-editor or even make rough approximations of the post-editing effort. Quality of an MT engine can be assessed on document- or on sentence-level. A tool to jointly provide all these functionalities does not exist yet. The overall objective of the project presented in this paper is to develop an MT quality assessment (MTQA) tool that simplifies the quality assessment of MT engines, combining quality evaluation and quality estimation on document- and sentence- level.

6. Final Devoicing in Dutch Medieval and Renaissance Texts: A Preliminary Study on Orthographic Variation

   Mirella De Sisto

   Filologia Germanica - Germanic Philology, pp. 99–117, Dec, 2022

   [Abs] [URL]

   This paper describes a preliminary quantitative study on orthographic variation of word-final devoiced consonants (e.g. altijd vs altijt ‘always’) in Dutch medieval and Renaissance texts. Five epic works in verse, either poems or plays, have been analysed, which cover a period from 1270 until 1637. The computational identification of orthographic marking of word-final devoicing, as well as the occurrence of instances without marking, allows, for each text under examination, to chart the different phases of the devoiced consonants’ spelling. The results show that the earlier medieval texts under investigation exhibit spelling which overtly displays devoiced consonants; later works, those written within the Renaissance period, show more instances of orthographic variation. Besides the preliminary results of this pilot study, this paper highlights the potential of applying similar computational approaches to larger data sets of texts.

2021

1. Automatic quantitative metrical analysis of Spanish Poetry with Rantanplan: a first approach

   Laura Hernández Lorenzo, Mirella De Sisto,  Pérez, Javier De la Rosa, Salvador Ros, and Elena González-Blanco

   Tackling the Toolkit. Plotting Poetry through Computational Literary Studies., 2021

   [Abs] [URL] [DOI]

   In this paper, we present a quantitative approach to Spanish poetry and versification through the application of our own automatic metrical tool, Rantanplan, to the complete poetic works of four Early Modern Spanish poets. The entire poetry by four representative authors of the period —Garcilaso de la Vega (1503-1536), Fernando de Herrera (1534-1597), Luis de Góngora (1561-1627), and Lope de Vega (1562-1635)—was automatically processed and stress positions were extracted. Thanks to the development of a new feature in Rantanplan for stanza identification, we could detect metrical structures as well. A quantitative analysis of stress positions, line lengths and stanzas used per author follows, aiming at modeling their complete metrical profile. Additionally, we present an application of the obtained metrical information for authorship attribution.

2. Generating Gender Augmented Data for NLP

   Nishtha Jain, Maja Popović, Declan Groves, and Eva Vanmassenhove

   In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pp. 93–102, Aug, 2021

   [Abs] [URL] [DOI]

   Gender bias is a frequent occurrence in NLP-based applications, especially pronounced in gender-inflected languages. Bias can appear through associations of certain adjectives and animate nouns with the natural gender of referents, but also due to unbalanced grammatical gender frequencies of inflected words. This type of bias becomes more evident in generating conversational utterances where gender is not specified within the sentence, because most current NLP applications still work on a sentence-level context. As a step towards more inclusive NLP, this paper proposes an automatic and generalisable re-writing approach for short conversational sentences. The rewriting method can be applied to sentences that, without extra-sentential context, have multiple equivalent alternatives in terms of gender. The method can be applied both for creating gender balanced outputs as well as for creating gender balanced training data. The proposed approach is based on a neural machine translation system trained to ‘translate’ from one gender alternative to another. Both the automatic and manual analysis of the approach show promising results with respect to the automatic generation of gender alternatives for conversational sentences in Spanish.

3. gENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena

   Eva Vanmassenhove, and Johanna Monti

   In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pp. 1–7, Aug, 2021

   [Abs] [URL] [DOI]

   Languages differ in terms of the absence or presence of gender features, the number of gender classes and whether and where gender features are explicitly marked. These cross-linguistic differences can lead to ambiguities that are difficult to resolve, especially for sentence-level MT systems. The identification of ambiguity and its subsequent resolution is a challenging task for which currently there aren’t any specific resources or challenge sets available. In this paper, we introduce gENder-IT, an English–Italian challenge set focusing on the resolution of natural gender phenomena by providing word-level gender tags on the English source side and multiple gender alternative translations, where needed, on the Italian target side.

4. Defining meaningful units. Challenges in sign segmentation and segment-meaning mapping (short paper)

   Mirella De Sisto, Dimitar Shterionov, Irene Murtagh, Myriam Vermeerbergen, and Lorraine Leeson

   In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pp. 98–103, Aug, 2021

   [Abs] [URL]

   This paper addresses the tasks of sign segmentation and segment-meaning mapping in the context of sign language (SL) recognition. It aims to give an overview of the linguistic properties of SL, such as coarticulation and simultaneity, which make these tasks complex. A better understanding of SL structure is the necessary ground for the design and development of SL recognition and segmentation methodologies, which are fundamental for machine translation of these languages. Based on this preliminary exploration, a proposal for mapping segments to meaning in the form of an agglomerate of lexical and non-lexical information is introduced.

5. Early-stage development of the SignON application and open framework – challenges and opportunities

   Dimitar Shterionov, John J O’Flaherty, Edward Keane, Connor O’Reilly, Marcello Paolo Scipioni, Marco Giovanelli, and Matteo Villa

   In Proceedings of Machine Translation Summit XVIII: Users and Providers Track, pp. 277–290, Aug, 2021

   [Abs] [URL]

   SignON is an EU Horizon 2020 Research and Innovation project, that is developing a smartphone application and an open framework to facilitate translation between different European sign, spoken and text languages. The framework will incorporate state of the art sign language recognition and presentation, speech processing technologies and, in its core, multi-modal, cross-language machine translation. The framework, dedicated to the computationally heavy tasks and distributed on the cloud powers the application – a lightweight app running on a standard mobile device. The application and framework are being researched, designed and developed through a co-creation user-centric approach with the European deaf and hard of hearing communities. In this session, the speakers will detail their progress, challenges and lessons learned in the early-stage development of the application and framework. They will also present their Agile DevOps approach and the next steps in the evolution of the SignON project.

6. NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender Neutral Alternatives

   Eva Vanmassenhove, Chris Emmery, and Dimitar Shterionov

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8940–8948, Nov, 2021

   [Abs] [URL] [DOI]

   Recent years have seen an increasing need for gender-neutral and inclusive language. Within the field of NLP, there are various mono- and bilingual use cases where gender inclusive language is appropriate, if not preferred due to ambiguity or uncertainty in terms of the gender of referents. In this work, we present a rule-based and a neural approach to gender-neutral rewriting for English along with manually curated synthetic data (WinoBias+) and natural data (OpenSubtitles and Reddit) benchmarks. A detailed manual and automatic evaluation highlights how our NeuTral Rewriter, trained on data generated by the rule-based approach, obtains word error rates (WER) below 0.18% on synthetic, in-domain and out-domain test sets.

7. Transformers analyzing poetry. Multilingual metrical pattern prediction with transfomer-based language models

   Javier De la Rosa,  Pérez, Mirella De Sisto, Laura Hernández Lorenzo, Aitor Diaz, Salvador Ros, and Elena González-Blanco

   Neural Computing and Applications, Nov, 2021

   [Abs] [URL] [DOI]

   The splitting of words into stressed and unstressed syllables is the foundation for the scansion of poetry, a process that aims at determining the metrical pattern of a line of verse within a poem. Intricate language rules and their exceptions, as well as poetic licenses exerted by the authors, make calculating these patterns a nontrivial task. Some rhetorical devices shrink the metrical length, while others might extend it. This opens the door for interpretation and further complicates the creation of automated scansion algorithms useful for automatically analyzing corpora on a distant reading fashion. In this paper, we compare the automated metrical pattern identification systems available for Spanish, English, and German, against fine-tuned monolingual and multilingual language models trained on the same task. Despite being initially conceived as models suitable for semantic tasks, our results suggest that transformers-based models retain enough structural information to perform reasonably well for Spanish on a monolingual setting, and outperforms both for English and German when using a model trained on the three languages, showing evidence of the benefits of cross-lingual transfer between the languages.

8. Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts

   Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck

   Computational Linguistics in the Netherlands Journal, vol. 11, pp. 213–230, Dec, 2021

   [Abs] [URL] [DOI]

   Continuously-growing data volumes lead to larger generic models. Specific use-cases are usually left out, since generic models tend to perform poorly in domain-specific cases. Our work addresses this gap with a method for selecting in-domain data from generic-domain (parallel text) corpora, for the task of machine translation. The proposed method ranks sentences in parallel general-domain data according to their cosine similarity with a monolingual domain-specific data set. We then select the top K sentences with the highest similarity score to train a new machine translation system tuned to the specific in-domain data. Our experimental results show that models trained on this in-domain data outperform models trained on generic or a mixture of generic and domain data. That is, our method selects high-quality domain-specific training instances at low computational cost and data size.