publications

2024

1. Zero-Shot vs. Translation-Based Cross-Lingual Transfer: The Case of Lexical Gaps

   Abteen Ebrahimi and Katharina von der Wense

   In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), Jun 2024

   Abs

   Cross-lingual transfer can be achieved through two main approaches: zero-shot transfer or machine translation (MT). While the former has been the dominant approach, both have been shown to be competitive. In this work, we compare the current performance and long-term viability of these methods. We leverage lexical gaps to create a multilingual question answering dataset, which provides a difficult domain for evaluation. Both approaches struggle in this setting, though zero-shot transfer performs better, as current MT outputs are not specific enough for the task. Using oracle translation offers the best performance, showing that this approach can perform well long-term, however current MT quality is a bottleneck. We also conduct an exploratory study to see if humans produce translations sufficient for the task with only general instructions. We find this to be true for the majority of translators, but not all. This indicates that while translation has the potential to outperform zero-shot approaches, creating MT models that generate accurate task-specific translations may not be straightforward.

2. Findings of the AmericasNLP 2024 Shared Task on Machine Translation into Indigenous Languages

   Abteen Ebrahimi, Ona Gibert, Raul Vazquez, and 8 more authors

   In Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024), Jun 2024

   Abs

   This paper presents the findings of the third iteration of the AmericasNLP Shared Task on Machine Translation. This year’s competition features eleven Indigenous languages found across North, Central, and South America. A total of six teams participate with a total of 157 submissions across all languages and models. Two baselines – the Sheffield and Helsinki systems from 2023 – are provided and represent hard-to-beat starting points for the competition. In addition to the baselines, teams are given access to a new repository of training data which consists of data collected by teams in prior shared tasks. Using ChrF++ as the main competition metric, we see improvements over the baseline for 4 languages: Chatino, Guarani, Quechua, and Rarámuri, with performance increases over the best baseline of 4.2 ChrF++. In this work, we present a summary of the submitted systems, results, and a human evaluation of system outputs for Bribri, which consists of both (1) a rating of meaning and fluency and (2) a qualitative error analysis of outputs from the best submitted system.

3. Findings of the AmericasNLP 2024 Shared Task on the Creation of Educational Materials for Indigenous Languages

   Luis Chiruzzo, Pavel Denisov, Alejandro Molina-Villegas, and 12 more authors

   In Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024), Jun 2024

   Abs

   This paper presents the results of the first shared task about the creation of educational materials for three indigenous languages of the Americas.The task proposes to automatically generate variations of sentences according to linguistic features that could be used for grammar exercises.The languages involved in this task are Bribri, Maya, and Guarani.Seven teams took part in the challenge, submitting a total of 22 systems, obtaining very promising results.

4. Since the Scientific Literature Is Multilingual, Our Models Should Be Too

   Abteen Ebrahimi and Kenneth Church

   In arXiv., Jun 2024
5. Academic Article Recommendation Using Multiple Perspectives

   Kenneth Church, Omar Alonso, Peter Vickers, and 3 more authors

   In arXiv., Jun 2024

2023

1. Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models

   Abteen Ebrahimi, Arya D. McCarthy, Arturo Oncevay, and 5 more authors

   In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, May 2023

   Abs

   Large multilingual models have inspired a new class of word alignment methods, which work well for the model’s pretraining languages. However, the languages most in need of automatic alignment are low-resource and, thus, not typically included in the pretraining data. In this work, we ask: How do modern aligners perform on unseen languages, and are they better than traditional methods? We contribute gold-standard alignments for Bribri–Spanish, Guarani–Spanish, Quechua–Spanish, and Shipibo-Konibo–Spanish. With these, we evaluate state-of-the-art aligners with and without model adaptation to the target language. Finally, we also evaluate the resulting alignments extrinsically through two downstream tasks: named entity recognition and part-of-speech tagging. We find that although transformer-based methods generally outperform traditional models, the two classes of approach remain competitive with each other.

2. Findings of the AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages

   Abteen Ebrahimi, Manuel Mager, Shruti Rijhwani, and 10 more authors

   In Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP), Jul 2023

   Abs

   In this work, we present the results of the AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages of the Americas. This edition of the shared task featured eleven language pairs, one of which – Chatino-Spanish – uses a newly collected evaluation dataset, consisting of professionally translated text from the legal domain. Seven teams participated in the shared task, with a total of 181 submissions. Additionally, we conduct a human evaluation of the best system outputs, and compare them to the best submissions from the prior shared task. We find that this analysis agrees with the quantitative measures used to rank submissions, which shows further improvements of 9.64 ChrF on average across all languages, when compared to the prior winning system.

2022

1. AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages

   Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, and 14 more authors

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2022

   Abs

   Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 Indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis-only models. We find that XLM-R’s zero-shot performance is poor for all 10 languages, with an average performance of 38.48%. Continued pretraining offers improvements, with an average accuracy of 43.85%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 49.12%.

2. Open-domain Dialogue Generation: What We Can Do, Cannot Do, and Should Do Next

   Katharina Kann, Abteen Ebrahimi, Joewie Koh, and 2 more authors

   In Proceedings of the 4th Workshop on NLP for Conversational AI, May 2022

   Abs

   Human–computer conversation has long been an interest of artificial intelligence and natural language processing research. Recent years have seen a dramatic improvement in quality for both task-oriented and open-domain dialogue systems, and an increasing amount of research in the area. The goal of this work is threefold: (1) to provide an overview of recent advances in the field of open-domain dialogue, (2) to summarize issues related to ethics, bias, and fairness that the field has identified as well as typical errors of dialogue systems, and (3) to outline important future challenges. We hope that this work will be of interest to both new and experienced researchers in the area.

3. Machine Translation Between High-resource Languages in a Language Documentation Setting

   Katharina Kann, Abteen Ebrahimi, Kristine Stenzel, and 1 more author

   In Proceedings of the First Workshop on NLP Applications to Field Linguistics, Oct 2022

   Abs

   Language documentation encompasses translation, typically into the dominant high-resource language in the region where the target language is spoken. To make data accessible to a broader audience, additional translation into other high-resource languages might be needed. Working within a project documenting Kotiria, we explore the extent to which state-of-the-art machine translation (MT) systems can support this second translation – in our case from Portuguese to English. This translation task is challenging for multiple reasons: (1) the data is out-of-domain with respect to the MT system’s training data, (2) much of the data is conversational, (3) existing translations include non-standard and uncommon expressions, often reflecting properties of the documented language, and (4) the data includes borrowings from other regional languages. Despite these challenges, existing MT systems perform at a usable level, though there is still room for improvement. We then conduct a qualitative analysis and suggest ways to improve MT between high-resource languages in a language documentation setting.

4. Findings of the Second AmericasNLP Competition on Speech-to-Text Translation

   Abteen Ebrahimi, Manuel Mager, Adam Wiemerslage, and 31 more authors

   In Proceedings of the NeurIPS 2022 Competitions Track, Oct 2022

   Abs

   Indigenous languages, including those from the Americas, have received very little attention from the machine learning (ML) and natural language processing (NLP) communities. To tackle the resulting lack of systems for these languages and the accompanying social inequalities affecting their speakers, we conduct the second AmericasNLP competition (and the first one in collaboration with NeurIPS), which is centered around speech-to-text translation systems for Indigenous languages of the Americas. The competition features three tasks – (1) automatic speech recognition, (2) text-based machine translation, and (3) speech-to-text translation – and two tracks: constrained and unconstrained. Five Indigenous languages are covered: Bribri, Guarani, Kotiria, Wa’ikhana, and Quechua. In this overview paper, we describe the tasks, tracks, and languages, introduce the baseline and participating systems, and end with a summary of ongoing and future challenges for the automatic translation of Indigenous languages.

5. AmericasNLI: Machine Translation and Natural Language Inference Systems for Indigenous Languages of the Americas

   Katharina Kann, Abteen Ebrahimi, Manuel Mager, and 15 more authors

   Frontiers in Artificial Intelligence, Oct 2022

2021

1. How to Adapt Your Pretrained Multilingual Model to 1600 Languages

   Abteen Ebrahimi and Katharina Kann

   In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Oct 2021
2. Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas

   Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, and 14 more authors

   In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, Oct 2021

2020

1. Athena: Constructing Dialogues Dynamically with Discourse Constraints.

   Vrindavan Harrison, Juraj Juraska, Wen Cui, and 11 more authors

   In arXiv., Oct 2020

2019

1. Curate and Generate: A Corpus and Method for Joint Control of Semantics and Style in Neural NLG

   Shereen Oraby, Vrindavan Harrison, Abteen Ebrahimi, and 1 more author

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Oct 2019