Publications

2024

Understanding poetry using natural language processing tools: a survey

De Sisto, Mirella, Hernández-Lorenzo, Laura, De la Rosa, Javier, Ros, Salvador, and González-Blanco, Elena

Digital Scholarship in the Humanities, pp. fqae001, Feb, 2024

[Abs] [URL] [DOI]

Analyzing poetry with automatic tools has great potential for improving verse-related research. Over the last few decades, this field has expanded notably and a large number of tools aiming at analyzing various aspects of poetry have been developed. However, the concrete connection between these tools and traditional scholars investigating poetry and metrics is often missing. The purpose of this article is to bridge this gap by providing a comprehensive survey of the automatic poetry analysis tools available for European languages. The tools are described and classified according to the language for which they are primarily developed, and to their functionalities and purpose. Particular attention is given to those that have open-source code or provide an online version with the same functionality. Combining more traditional research with these tools has clear advantages: it provides the opportunity to address theoretical questions with the support of large amounts of data; also, it allows for the development of new and diversified approaches.

2023

Evaluating the Effectiveness of Pre-trained Language Models in Predicting the Helpfulness of Online Product Reviews

Boluki, Ali, Pourmostafa Roshan Sharami, Javad, and Shterionov, Dimitar

2023

[Abs] [URL]

Businesses and customers can gain valuable information from product reviews. The sheer number of reviews often necessitates ranking them based on their potential helpfulness. However, only a few reviews ever receive any helpfulness votes on online marketplaces. Sorting all reviews based on the few existing votes can cause helpful reviews to go unnoticed because of the limited attention span of readers. The problem of review helpfulness prediction is even more important for higher review volumes, and newly written reviews or launched products. In this work we compare the use of RoBERTa and XLM-R language models to predict the helpfulness of online product reviews. The contributions of our work in relation to literature include extensively investigating the efficacy of state-of-the-art language models – both monolingual and multilingual – against a robust baseline, taking ranking metrics into account when assessing these approaches, and assessing multilingual models for the first time. We employ the Amazon review dataset for our experiments. According to our study on several product categories, multilingual and monolingual pre-trained language models outperform the baseline that utilizes random forest with handcrafted features as much as 23% in RMSE. Pre-trained language models reduce the need for complex text feature engineering. However, our results suggest that pre-trained multilingual models may not be used for fine-tuning only one language. We assess the performance of language models with and without additional features. Our results show that including additional features like product rating by the reviewer can further help the predictive methods.
SignON: Sign Language Translation. Progress and challenges

Vandeghinste, Vincent, Shterionov, Dimitar, De Sisto, Mirella, Brady, Aoife, De Coster, Mathieu, Leeson, Lorraine, Blat, Josep, Picron, Frankie, Scipioni, Marcello Paolo, Parikh, Aditya, Bosch, Louis, O’Flaherty, John, Dambre, Joni, Rijckaert, Jorn, Vanroy, Bram, Nogales, Victor Ubieto, Gomez, Santiago Egea, Schuurman, Ineke, Labaka, Gorka, Nunez-Marcos, Adrian, Murtagh, Irene, McGill, Euan, and Saggion, Horacio

In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 2023

[URL]
GoSt-ParC-Sign: Gold Standard Parallel Corpus of Sign and spoken language

De Sisto, Mirella, Vandeghinste, Vincent, Soetemans, Lien, Brosens, Caro, and Shterionov, Dimitar

In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 503–504, 2023

[URL]
Report on Europe’s Sign Languages. Deliverable European Language Equality II project

Vandeghinste, Vincent, De Sisto, Mirella, Kopf, Maria, Schulder, Marc, Brosens, Caro, Soetemans, Lien, Omardeen, Rehana, Picron, Frankie, Landuyt, Davy Van, Murtagh, Irene, Avramidis, Eleftherios, and De Coster, Mathieu

2023

[URL]
NGT-HoReCo and GoSt-ParC-Sign: Two new Sign Language - Spoken Language parallel corpora

De Sisto, Mirella, Vandeghinste, Vincent, Shterionov, Dimitar, Soetemans, Lien, and Brosens, Caro

In CLARIN Annual Conference Proceedings, 2023, pp. 6–9, 2023

[URL]
The development of a poetic tradition. A study on a Dutch Renaissance Poetry Corpus

De Sisto, Mirella

Studia Metrica et Poetica, vol. 10, pp. 36–68, 2023

[URL] [DOI]
Tailoring Domain Adaptation for Machine Translation Quality Estimation

Sharami, Javad Pourmostafa Roshan, Shterionov, Dimitar, Blain, Frédéric, Vanmassenhove, Eva, De Sisto, Mirella, Emmery, Chris, and Spronck, Pieter

In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 9–20, Jun, 2023

[Abs] [URL]

While quality estimation (QE) can play an important role in the translation process, its effectiveness relies on the availability and quality of training data. For QE in particular, high-quality labeled data is often lacking due to the high-cost and effort associated with labeling such data. Aside from the data scarcity challenge, QE models should also be generalizabile, i.e., they should be able to handle data from different domains, both generic and specific. To alleviate these two main issues — data scarcity and domain mismatch — this paper combines domain adaptation and data augmentation within a robust QE system. Our method is to first train a generic QE model and then fine-tune it on a specific domain while retaining generic knowledge. Our results show a significant improvement for all the language pairs investigated, better cross-lingual inference, and a superior performance in zero-shot learning scenarios as compared to state-of-the-art baselines.
A New English-Dutch-NGT Corpus for the Hospitality Domain

De Sisto, Mirella, Vandeghinste, Vincent, and Shterionov, Dimitar

In Proceedings of the Second International Workshop on Automatic Translation for Signed and Spoken Languages, pp. 34–37, Jun, 2023

[Abs] [URL]

One of the major challenges hampering the development of language technology which targets sign languages is the extremely limited availability of good quality data geared towards machine learning and deep learning approaches. In this paper we introduce the NGT-Dutch Hotel Review Corpus (NGT-HoReCo), which addresses this issue by providing multimodal parallel data in English, Dutch and Sign Language of the Netherlands (NGT). The corpus contains 283 hotel reviews in written English, translated into written Dutch and into NGT videos. It will be made publicly available through CLARIN and through the ELG platform.
Authorship Attribution of Late 19th Century Novels using GAN-BERT

Silva, Kanishka, Can, Burcu, Blain, Frédéric, Sarwar, Raheem, Ugolini, Laura, and Mitkov, Ruslan

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 310–320, Jul, 2023

[Abs] [URL] [DOI]

Authorship attribution aims to identify the author of an anonymous text. The task becomes even more worthwhile when it comes to literary works. For example, pen names were commonly used by female authors in the 19th century resulting in some literary works being incorrectly attributed or claimed. With this motivation, we collated a dataset of late 19th century novels in English. Due to the imbalance in the dataset and the unavailability of enough data per author, we employed the GANBERT model along with data sampling strategies to fine-tune a transformer-based model for authorship attribution. Differently from the earlier studies on the GAN-BERT model, we conducted transfer learning on comparatively smaller author subsets to train more focused author-specific models yielding performance over 0.88 accuracy and F1 scores. Furthermore, we observed that increasing the sample size has a negative impact on the model’s performance. Our research mainly contributes to the ongoing authorship attribution research using GAN-BERT architecture, especially in attributing disputed novelists in the late 19th century.
Text Data Augmentation Using Generative Adversarial Networks – A Systematic Review

Silva, Kanishka, Can, Burcu, Sarwar, Raheem, Blain, Frédéric, and Mitkov, Ruslan

Journal of Computational and Applied Linguistics, vol. 1, pp. 6–38, Jul, 2023

[Abs] [DOI]

Insufficient data is one of the main drawbacks in natural language processing tasks, and the most prevalent solution is to collect a decent amount of data that will be enough for the optimisation of the model. However, recent research directions are strategically moving towards increasing training examples due to the nature of the data-hungry neural models. Data augmentation is an emerging area that aims to ensure the diversity of data without attempting to collect new data exclusively to boost a model’s performance. Limitations in data augmentation, especially for textual data, are mainly due to the nature of language data, which is precisely discrete. Generative Adversarial Networks (GANs) were initially introduced for computer vision applications, aiming to generate highly realistic images by learning the image representations. Recent research has focused on using GANs for text generation and augmentation. This systematic review aims to present the theoretical background of GANs and their use for text augmentation alongside a systematic review of recent textual data augmentation applications such as sentiment analysis, low resource language generation, hate speech detection and fraud review analysis. Further, a notion of challenges in current research and future directions of GAN-based text augmentation are discussed in this paper to pave the way for researchers especially working on low-text resources.
Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent

Freitag, Markus, Mathur, Nitika, Lo, Chi-kiu, Avramidis, Eleftherios, Rei, Ricardo, Thompson, Brian, Kocmi, Tom, Blain, Frédéric, Deutsch, Daniel, Stewart, Craig, Zerva, Chrysoula, Castilho, Sheila, Lavie, Alon, and Foster, George

In Proceedings of the Eighth Conference on Machine Translation, pp. 578–628, Dec, 2023

[Abs] [URL] [DOI]

This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year’s success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks. We present an extensive analysis on how well metrics perform on three language pairs: Chinese-English, Hebrew-English on the sentence-level and English-German on the paragraph-level. The results strongly confirm the results reported last year, that neural-based metrics are significantly better than non-neural metrics in their levels of correlation with human judgments. Further, we investigate the impact of bad reference translations on the correlations of metrics with human judgment. We present a novel approach for generating synthetic reference translations based on the collection of MT system outputs and their corresponding MQM ratings, which has the potential to mitigate bad reference issues we observed this year for some language pairs. Finally, we also study the connections between the magnitude of metric differences and their expected significance in human evaluation, which should help the community to better understand and adopt new metrics.
Findings of the WMT 2023 Shared Task on Quality Estimation

Blain, Frédéric, Zerva, Chrysoula, Ribeiro, Ricardo, Guerreiro, Nuno M., Kanojia, Diptesh, Souza, José G., Silva, Beatriz, Vaz, Tânia, Jingxuan, Yan, Azadi, Fatemeh, Orasan, Constantin, and Martins, André

In Proceedings of the Eighth Conference on Machine Translation, pp. 629–653, Dec, 2023

[Abs] [URL] [DOI]

We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test-set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions.

2022

The Ecological Footprint of Neural Machine Translation Systems

Shterionov, Dimitar, and Vanmassenhove, Eva

2022

[URL] [DOI]
Modelli di demarcazione della metà verso nel metro rinascimentale romanzo

De Sisto, Mirella

ILLA - Nuove Ricerche Umanistiche, vol. Interruzioni e cesure. Fenomeni e pratiche della discontinuità in linguistica, letteratura e arti performative, pp. 192–200, 2022

[Abs] [URL]

The present paper discusses two types of mid-line marking attested in Romance traditions instantiations of Renaissance metre and outlines an explanation for their structural differences. On the one hand, some poetic traditions, namely, French and Catalan, preserved the caesura of the source form. On the other hand, Italian, Spanish and Portuguese poetry developed a kind of mid-line marking which is based on the obligatory prominence of a metrical position in the middle of the line. A further distinction can be made when considering that French and Catalan caesurae are not exactly the same: in French, this plays the role of the only metrical element smaller than the line; ln Catalan, instead, a prominent position line-medially is also dividing the line in two hemistiches. Catalan caesura constitutes a hybrid form between a proper caesura, in the French way, and a prominence mid-line marking system.
Quality Estimation for the Translation Industry – Data Challenges

Pourmostafa Roshan Sharami, Javad, Murgolo, Elena, and Shterionov, Dimitar

In The 32nd Meeting of Computational Linguistics in The Netherlands, CLIN, Jun, 2022

[Abs] [URL] [DOI]

Machine Translation (MT) has become an irreplaceable part of translation industry workflows. With a direct impact on productivity, it is very important for human post-editors and project managers to be informed about the translation quality of MT.MT Quality estimation (QE) is the task of predicting the quality of a translation without human references. For the translation industry QE can be used as an indicator of the amount of post-editing needed as well as of productivity. QE can be applied at word-, sentence- or document-level. In the cases of sentence- and document-level QE, given a source text and its MT counterpart the task is to predict a score (typically TER) that indicates the translation quality. As with most NLP tasks nowadays, state-of-the-art QE has been achieved using DL methods. Optimal QE performance is not only a question of model architecture and hyperparameters but it strongly depends on the quantity and quality of the data.In a business environment, such as the one of the translation industry models or system that are used in production should adhere to economic and usability criteria. QE for the translation industry should be optimised for domain- and use-case specific data, should be efficient and be adaptable. In a collaborative project between Orbital14 and Tilburg University, funded by Aglatech14, we develop a framework for MT quality assessment (MTQA) which strongly relies on QE. In this project we focus on a specific domain (patents, IP) and a language pair (English-Italian).In this work we present our approach to data collection, analysis and preprocessing prior to building QE models. We started with proprietary data provided by Aglatech14 to identify specific patterns that need to be covered by the QE tool. As the volume of initial data was not sufficient to build robust DL models (42K source-translation-post-edit triplets, from which we compute the TER scores) we added a corpus of 127K source-human translation sentences. We used Aglatech14’s translation model to translate the source and generate a pseudo corpus of triplets (source-MT-post-edit), which were then used to compute TER scores. To further extend our data set we used publicly available data – a generic English- Italian corpus containing 105M sentence pairs. Aiming at a domain-specific QE, we select only similar to the industry data sentence pairs. We used a ranking data selection method based on cosine similarity to selected additional 42K sentences – those with the highest similarity score to the Aglatech14 data. After data selection, wetranslated the selected source data to create new synthetic data comprising of source, machine translation and target sentences. We consider the target sentences as post-edited sentences. As in the previous case, we computed the TER score between the MT and the target to use as labels for training our QE models.We use these data to build state-of-the-art QE models and evaluate their performance against a gold standard reference set, provided by Aglatech14.
Challenges with Sign Language Datasets for Sign Language Recognition and Translation

De Sisto, Mirella, Vandeghinste, Vincent, Egea Gómez, Santiago, De Coster, Mathieu, Shterionov, Dimitar, and Saggion, Horacio

In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2478–2487, Jun, 2022

[Abs] [URL]

Sign Languages (SLs) are the primary means of communication for at least half a million people in Europe alone. However, the development of SL recognition and translation tools is slowed down by a series of obstacles concerning resource scarcity and standardization issues in the available data. The former challenge relates to the volume of data available for machine learning as well as the time required to collect and process new data. The latter obstacle is linked to the variety of the data, i.e., annotation formats are not unified and vary amongst different resources. The available data formats are often not suitable for machine learning, obstructing the provision of automatic tools based on neural models. In the present paper, we give an overview of these challenges by comparing various SL corpora and SL machine learning datasets. Furthermore, we propose a framework to address the lack of standardization at format level, unify the available resources and facilitate SL research for different languages. Our framework takes ELAN files as inputs and returns textual and visual data ready to train SL recognition and translation models. We present a proof of concept, training neural translation models on the data produced by the proposed framework.
A Quality Estimation and Quality Evaluation Tool for the Translation Industry

Murgolo, Elena, Sharami, Javad Pourmostafa Roshan, and Shterionov, Dimitar

In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pp. 307–308, Jun, 2022

[Abs] [URL]

With the increase in machine translation (MT) quality over the latest years, it has now become a common practice to integrate MT in the workflow of language service providers (LSPs) and other actors in the translation industry. With MT having a direct impact on the translation workflow, it is important not only to use high-quality MT systems, but also to understand the quality dimension so that the humans involved in the translation workflow can make informed decisions. The evaluation and monitoring of MT output quality has become one of the essential aspects of language technology management in LSPs’ workflows. First, a general practice is to carry out human tests to evaluate MT output quality before deployment. Second, a quality estimate of the translated text, thus after deployment, can inform post editors or even represent post-editing effort. In the former case, based on the quality assessment of a candidate engine, an informed decision can be made whether the engine would be deployed for production or not. In the latter, a quality estimate of the translation output can guide the human post-editor or even make rough approximations of the post-editing effort. Quality of an MT engine can be assessed on document- or on sentence-level. A tool to jointly provide all these functionalities does not exist yet. The overall objective of the project presented in this paper is to develop an MT quality assessment (MTQA) tool that simplifies the quality assessment of MT engines, combining quality evaluation and quality estimation on document- and sentence- level.
Final Devoicing in Dutch Medieval and Renaissance Texts: A Preliminary Study on Orthographic Variation

De Sisto, Mirella

Filologia Germanica - Germanic Philology, pp. 99–117, Dec, 2022

[Abs] [URL]

This paper describes a preliminary quantitative study on orthographic variation of word-final devoiced consonants (e.g. altijd vs altijt ‘always’) in Dutch medieval and Renaissance texts. Five epic works in verse, either poems or plays, have been analysed, which cover a period from 1270 until 1637. The computational identification of orthographic marking of word-final devoicing, as well as the occurrence of instances without marking, allows, for each text under examination, to chart the different phases of the devoiced consonants’ spelling. The results show that the earlier medieval texts under investigation exhibit spelling which overtly displays devoiced consonants; later works, those written within the Renaissance period, show more instances of orthographic variation. Besides the preliminary results of this pilot study, this paper highlights the potential of applying similar computational approaches to larger data sets of texts.

2021

Automatic quantitative metrical analysis of Spanish Poetry with Rantanplan: a first approach

Hernández Lorenzo, Laura, De Sisto, Mirella, Pérez, , De la Rosa, Javier, Ros, Salvador, and González-Blanco, Elena

Tackling the Toolkit. Plotting Poetry through Computational Literary Studies., 2021

[Abs] [URL] [DOI]

In this paper, we present a quantitative approach to Spanish poetry and versification through the application of our own automatic metrical tool, Rantanplan, to the complete poetic works of four Early Modern Spanish poets. The entire poetry by four representative authors of the period —Garcilaso de la Vega (1503-1536), Fernando de Herrera (1534-1597), Luis de Góngora (1561-1627), and Lope de Vega (1562-1635)—was automatically processed and stress positions were extracted. Thanks to the development of a new feature in Rantanplan for stanza identification, we could detect metrical structures as well. A quantitative analysis of stress positions, line lengths and stanzas used per author follows, aiming at modeling their complete metrical profile. Additionally, we present an application of the obtained metrical information for authorship attribution.
Generating Gender Augmented Data for NLP

Jain, Nishtha, Popović, Maja, Groves, Declan, and Vanmassenhove, Eva

In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pp. 93–102, Aug, 2021

[Abs] [URL] [DOI]

Gender bias is a frequent occurrence in NLP-based applications, especially pronounced in gender-inflected languages. Bias can appear through associations of certain adjectives and animate nouns with the natural gender of referents, but also due to unbalanced grammatical gender frequencies of inflected words. This type of bias becomes more evident in generating conversational utterances where gender is not specified within the sentence, because most current NLP applications still work on a sentence-level context. As a step towards more inclusive NLP, this paper proposes an automatic and generalisable re-writing approach for short conversational sentences. The rewriting method can be applied to sentences that, without extra-sentential context, have multiple equivalent alternatives in terms of gender. The method can be applied both for creating gender balanced outputs as well as for creating gender balanced training data. The proposed approach is based on a neural machine translation system trained to ‘translate’ from one gender alternative to another. Both the automatic and manual analysis of the approach show promising results with respect to the automatic generation of gender alternatives for conversational sentences in Spanish.
gENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena

Vanmassenhove, Eva, and Monti, Johanna

In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pp. 1–7, Aug, 2021

[Abs] [URL] [DOI]

Languages differ in terms of the absence or presence of gender features, the number of gender classes and whether and where gender features are explicitly marked. These cross-linguistic differences can lead to ambiguities that are difficult to resolve, especially for sentence-level MT systems. The identification of ambiguity and its subsequent resolution is a challenging task for which currently there aren’t any specific resources or challenge sets available. In this paper, we introduce gENder-IT, an English–Italian challenge set focusing on the resolution of natural gender phenomena by providing word-level gender tags on the English source side and multiple gender alternative translations, where needed, on the Italian target side.
Defining meaningful units. Challenges in sign segmentation and segment-meaning mapping (short paper)

De Sisto, Mirella, Shterionov, Dimitar, Murtagh, Irene, Vermeerbergen, Myriam, and Leeson, Lorraine

In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pp. 98–103, Aug, 2021

[Abs] [URL]

This paper addresses the tasks of sign segmentation and segment-meaning mapping in the context of sign language (SL) recognition. It aims to give an overview of the linguistic properties of SL, such as coarticulation and simultaneity, which make these tasks complex. A better understanding of SL structure is the necessary ground for the design and development of SL recognition and segmentation methodologies, which are fundamental for machine translation of these languages. Based on this preliminary exploration, a proposal for mapping segments to meaning in the form of an agglomerate of lexical and non-lexical information is introduced.
Early-stage development of the SignON application and open framework – challenges and opportunities

Shterionov, Dimitar, J O’Flaherty, John, Keane, Edward, O’Reilly, Connor, Paolo Scipioni, Marcello, Giovanelli, Marco, and Villa, Matteo

In Proceedings of Machine Translation Summit XVIII: Users and Providers Track, pp. 277–290, Aug, 2021

[Abs] [URL]

SignON is an EU Horizon 2020 Research and Innovation project, that is developing a smartphone application and an open framework to facilitate translation between different European sign, spoken and text languages. The framework will incorporate state of the art sign language recognition and presentation, speech processing technologies and, in its core, multi-modal, cross-language machine translation. The framework, dedicated to the computationally heavy tasks and distributed on the cloud powers the application – a lightweight app running on a standard mobile device. The application and framework are being researched, designed and developed through a co-creation user-centric approach with the European deaf and hard of hearing communities. In this session, the speakers will detail their progress, challenges and lessons learned in the early-stage development of the application and framework. They will also present their Agile DevOps approach and the next steps in the evolution of the SignON project.
NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender Neutral Alternatives

Vanmassenhove, Eva, Emmery, Chris, and Shterionov, Dimitar

In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8940–8948, Nov, 2021

[Abs] [URL] [DOI]

Recent years have seen an increasing need for gender-neutral and inclusive language. Within the field of NLP, there are various mono- and bilingual use cases where gender inclusive language is appropriate, if not preferred due to ambiguity or uncertainty in terms of the gender of referents. In this work, we present a rule-based and a neural approach to gender-neutral rewriting for English along with manually curated synthetic data (WinoBias+) and natural data (OpenSubtitles and Reddit) benchmarks. A detailed manual and automatic evaluation highlights how our NeuTral Rewriter, trained on data generated by the rule-based approach, obtains word error rates (WER) below 0.18% on synthetic, in-domain and out-domain test sets.
Transformers analyzing poetry. Multilingual metrical pattern prediction with transfomer-based language models

De la Rosa, Javier, Pérez, , De Sisto, Mirella, Hernández Lorenzo, Laura, Diaz, Aitor, Ros, Salvador, and González-Blanco, Elena

Neural Computing and Applications, Nov, 2021

[Abs] [URL] [DOI]

The splitting of words into stressed and unstressed syllables is the foundation for the scansion of poetry, a process that aims at determining the metrical pattern of a line of verse within a poem. Intricate language rules and their exceptions, as well as poetic licenses exerted by the authors, make calculating these patterns a nontrivial task. Some rhetorical devices shrink the metrical length, while others might extend it. This opens the door for interpretation and further complicates the creation of automated scansion algorithms useful for automatically analyzing corpora on a distant reading fashion. In this paper, we compare the automated metrical pattern identification systems available for Spanish, English, and German, against fine-tuned monolingual and multilingual language models trained on the same task. Despite being initially conceived as models suitable for semantic tasks, our results suggest that transformers-based models retain enough structural information to perform reasonably well for Spanish on a monolingual setting, and outperforms both for English and German when using a model trained on the three languages, showing evidence of the benefits of cross-lingual transfer between the languages.
Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts

Pourmostafa Roshan Sharami, Javad, Shterionov, Dimitar, and Spronck, Pieter

Computational Linguistics in the Netherlands Journal, vol. 11, pp. 213–230, Dec, 2021

[Abs] [URL] [DOI]

Continuously-growing data volumes lead to larger generic models. Specific use-cases are usually left out, since generic models tend to perform poorly in domain-specific cases. Our work addresses this gap with a method for selecting in-domain data from generic-domain (parallel text) corpora, for the task of machine translation. The proposed method ranks sentences in parallel general-domain data according to their cosine similarity with a monolingual domain-specific data set. We then select the top K sentences with the highest similarity score to train a new machine translation system tuned to the specific in-domain data. Our experimental results show that models trained on this in-domain data outperform models trained on generic or a mixture of generic and domain data. That is, our method selects high-quality domain-specific training instances at low computational cost and data size.