Published in Proceedings of the third workshop on NLP for computer-assisted language learning, 2014
Recommended citation: Nikolina Koleva, Andrea Horbach, Alexis Palmer, Simon Ostermann, and Manfred Pinkal. (2014). "Paraphrase Detection for Short Answer Scoring." Proceedings of the third workshop on NLP for computer-assisted language learning.
Download Paper
Published in Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications, 2015
Recommended citation: Simon Ostermann, Andrea Horbach, and Manfred Pinkal. (2015). "Annotating Entailment Relations for Shortanswer Questions." Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications.
Download Paper
Published in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), 2016
This paper presents the InScript corpus (Narrative Texts Instantiating Script structure). InScript is a corpus of 1,000 stories centered around 10 different scenarios. Verbs and noun phrases are annotated with event and participant types, respectively. Additionally, the text is annotated with coreference information. The corpus shows rich lexical variation and will serve as a unique resource for the study of the role of script knowledge in natural language processing.
Recommended citation: Ashutosh Modi, Tatjana Anikina, Simon Ostermann, and Manfred Pinkal. (2016). "InScript: Narrative texts annotated with script information." Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).
Download Paper
Published in Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), 2017
Script knowledge plays a central role in text understanding and is relevant for a variety of downstream tasks. In this paper, we consider two recent datasets which provide a rich and general representation of script events in terms of paraphrase sets. We introduce the task of mapping event mentions in narrative texts to such script event types, and present a model for this task that exploits rich linguistic representations as well as information on temporal ordering. The results of our experiments demonstrate that this complex task is indeed feasible.
Recommended citation: Simon Ostermann, Michael Roth, Stefan Thater, and Manfred Pinkal. (2017). "Aligning Script Events with Narrative Texts." Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017).
Download Paper
Published in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
Recommended citation: Simon Ostermann, Hannah Seitz, Stefan Thater, and Manfred Pinkal. (2018). "Mapping Texts to Scripts: An Entailment Study." Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Download Paper
Published in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
Recommended citation: Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. (2018). "MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge." Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Download Paper
Published in Proceedings of the 12th International Workshop on Semantic Evaluation, 2018
This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95\%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98\%.
Recommended citation: Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. (2018). "SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge." Proceedings of the 12th International Workshop on Semantic Evaluation.
Download Paper
Published in Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), 2019
We introduce MCScript2.0, a machine comprehension corpus for the end-to-end evaluation of script knowledge. MCScript2.0 contains approx. 20,000 questions on approx. 3,500 texts, crowdsourced based on a new collection process that results in challenging questions. Half of the questions cannot be answered from the reading texts, but require the use of commonsense and, in particular, script knowledge. We give a thorough analysis of our corpus and show that while the task is not challenging to humans, existing machine comprehension models fail to perform well on the data, even if they make use of a commonsense knowledge base. The dataset is available at http://www.sfb1102.uni-saarland.de/?page_id=2582
Recommended citation: Simon Ostermann, Michael Roth, and Manfred Pinkal. (2019). "MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants." Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019).
Download Paper
Published in Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 2019
This paper reports on the results of the shared tasks of the COIN workshop at EMNLP-IJCNLP 2019. The tasks consisted of two machine comprehension evaluations, each of which tested a system’s ability to answer questions/queries about a text. Both evaluations were designed such that systems need to exploit commonsense knowledge, for example, in the form of inferences over information that is available in the common ground but not necessarily mentioned in the text. A total of five participating teams submitted systems for the shared tasks, with the best submitted system achieving 90.6\% accuracy and 83.7\% F1-score on task 1 and task 2, respectively.
Recommended citation: Simon Ostermann, Sheng Zhang, Michael Roth, and Peter Clark. (2019). "Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report." Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing.
Download Paper
Published in Saarländische Universitäts- und Landesbibliothek, 2020
While people process text, they make frequent use of information that is assumed to be common ground and left implicit in the text. One important type of such commonsense knowledge is script knowledge, which is the knowledge about the events and participants in everyday activities such as visiting a restaurant. Due to its implicitness, it is hard for machines to exploit such script knowledge for natural language processing (NLP). This dissertation addresses the role of script knowledge in a central field of NLP, natural language understanding (NLU). In the first part of this thesis, we address script parsing. The idea of script parsing is to align event and participant mentions in a text with an underlying script representation. This makes it possible for a system to leverage script knowledge for downstream tasks. We develop the first script parsing model for events that can be trained on a large scale on crowdsourced script data. The model is implemented as a linear-chain conditional random field and trained on sequences of short event descriptions, implicitly exploiting the inherent event ordering information. We show that this ordering information plays a crucial role for script parsing. Our model provides an important first step towards facilitating the use of script knowledge for NLU. In the second part of the thesis, we move our focus to an actual application in the area of NLU, i.e. machine comprehension. For the first time, we provide data sets for the systematic evaluation of the contribution of script knowledge for machine comprehension. We create MCScript, a corpus of narrations about everyday activities and questions on the texts. By collecting questions based on a scenario rather than a text, we aimed at creating challenging questions which require script knowledge for finding the correct answer. Based on the findings of a shared task carried out with the data set, which indicated that script knowledge is not relevant for good performance on our corpus, we revised the data collection process and created a second version of the data set.
Recommended citation: Simon Ostermann. (2020). "Script Knowledge for Natural Language Understanding." Saarländische Universitäts- und Landesbibliothek.
Download Paper
Published in Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2023
Pretrained language models (PLMs) form the basis of most state-of-the-art NLP technologies. Nevertheless, they are essentially black boxes: Humans do not have a clear understanding of what knowledge is encoded in different parts of the models, especially in individual neurons. A contrast is in computer vision, where feature visualization provides a decompositional interpretability technique for neurons of vision models. Activation maximization is used to synthesize inherently interpretable visual representations of the information encoded in individual neurons. Our work is inspired by this but presents a cautionary tale on the interpretability of single neurons, based on the first large-scale attempt to adapt activation maximization to NLP, and, more specifically, large PLMs. We propose feature textualization, a technique to produce dense representations of neurons in the PLM word embedding space. We apply feature textualization to the BERT model to investigate whether the knowledge encoded in individual neurons can be interpreted and symbolized. We find that the produced representations can provide insights about the knowledge encoded in individual neurons, but that individual neurons do not represent clear-cut symbolic units of language such as words. Additionally, we use feature textualization to investigate how many neurons are needed to encode words in BERT.
Recommended citation: Tanja Baeumel, Soniya Vijayakumar, Josef van Genabith, Guenter Neumann, and Simon Ostermann. (2023). "Investigating the Encoding of Words in BERT's Neurons Using Feature Textualization." Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP.
Download Paper
Published in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023
In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, “oil” mixed with “salt” and “pepper” could later be referred to as a “mixture”. Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression “mixture”, and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as Visual-linguistic Ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of visual-linguistic ambiguity, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization with one joint model in multitask learning, as compared to a strong single-task baseline.
Recommended citation: Cennet Oguz, Pascal Denis, Emmanuel Vincent, Simon Ostermann, and Josef van Genabith. (2023). "Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Download Paper
Published in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024
The Common European Language Data Space (LDS) is an integral part of the EU data strategy, which aims at developing a single market for data. Its decentralised technical infrastructure and governance scheme are currently being developed by the LDS project, which also has dedicated tasks for proof-of-concept prototypes, handling legal aspects, raising awareness and promoting the LDS through events and social media channels. The LDS is part of a broader vision for establishing all necessary components to develop European large language models.
Recommended citation: Georg Rehm, Stelios Piperidis, Khalid Choukri, Andrejs Vasiļjevs, Katrin Marheinecke, Victoria Arranz, Aivars Bērziņš, Miltos Deligiannis, Dimitris Galanis, Maria Giagkou, Katerina Gkirtzou, Dimitris Gkoumas, Annika Grützner-Zahn, Athanasia Kolovou, Penny Labropoulou, Andis Lagzdiņš, Elena Leitner, Valérie Mapelli, Hélène Mazo, Simon Ostermann, Stefania Racioppa, Mickaël Rigault, and Leon Voukoutis. (2024). "Common European Language Data Space." Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).
Download Paper
Published in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop), 2024
Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks. These models undergo two stages of training: pretraining on a huge corpus of data and fine-tuning on a specific downstream task. The pretraining phase is extremely compute-intensive and requires several high-performance computing devices like GPUs and several days or even months of training, but it is crucial for the model to capture global knowledge and also has a significant impact on the fine-tuning task. This is a major roadblock for researchers without access to sophisticated computing resources. To overcome this challenge, we propose two novel hybrid architectures called HybridBERT (HBERT), which combine self-attention and additive attention mechanisms together with sub-layer normalization. We introduce a computing budget to the pretraining phase, limiting the training time and usage to a single GPU. We show that HBERT attains twice the pretraining accuracy of a vanilla-BERT baseline. We also evaluate our proposed models on two downstream tasks, where we outperform BERT-base while accelerating inference. Moreover, we study the effect of weight initialization with a limited pretraining budget. The code and models are publicly available at: www.github.com/gokulsg/HBERT/.
Recommended citation: Gokul Srinivasagan and Simon Ostermann. (2024). "HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop).
Download Paper
Published in Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), 2024
This paper explores the integration of graph knowledge from linguistic ontologies into multilingual Large Language Models (LLMs) using adapters to improve performance for low-resource languages (LRLs) in sentiment analysis (SA) and named entity recognition (NER). Building upon successful parameter-efficient fine-tuning techniques, such as K-ADAPTER and MAD-X, we propose a similar approach for incorporating knowledge from multilingual graphs, connecting concepts in various languages with each other through linguistic relationships, into multilingual LLMs for LRLs. Specifically, we focus on eight LRLs — Maltese, Bulgarian, Indonesian, Nepali, Javanese, Uyghur, Tibetan, and Sinhala — and employ language-specific adapters fine-tuned on data extracted from the language-specific section of ConceptNet, aiming to enable knowledge transfer across the languages covered by the knowledge graph. We compare various fine-tuning objectives, including standard Masked Language Modeling (MLM), MLM with full-word masking, and MLM with targeted masking, to analyze their effectiveness in learning and integrating the extracted graph data. Through empirical evaluation on language-specific tasks, we assess how structured graph knowledge affects the performance of multilingual LLMs for LRLs in SA and NER, providing insights into the potential benefits of adapting language models for low-resource scenarios.
Recommended citation: Daniil Gurgurov, Mareike Hartmann, and Simon Ostermann. (2024). "Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters." Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024).
Download Paper
Published in Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), 2024
Recommended citation: Beso Mikaberidze, Temo Saghinadze, Guram Mikaberidze, Raphael Kalandadze, Konstantine Pkhakadze, Josef van Genabith, Simon Ostermann, Lonneke van der Plas, and Philipp Müller. (2024). "A Comparison of Different Tokenization Methods for the Georgian Language." Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024).
Download Paper
Published in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024
Multilingual anaphora resolution identifies referring expressions and implicit arguments in texts and links to antecedents that cover several languages. In the most challenging setting, cross-lingual anaphora resolution, training data, and test data are in different languages. As knowledge needs to be transferred across languages, this task is challenging, both in the multilingual and cross-lingual setting. We hypothesize that one way to alleviate some of the difficulty of the task is to include multimodal information in the form of images (i.e. frames extracted from instructional videos). Such visual inputs are by nature language agnostic, therefore cross- and multilingual anaphora resolution should benefit from visual information. In this paper, we provide the first multilingual and multimodal dataset annotated with anaphoric relations and present experimental results for end-to-end multimodal and multilingual anaphora resolution. Given gold mentions, multimodal features improve anaphora resolution results by \textbackslashtextasciitilde10 \% for unseen languages.
Recommended citation: Cennet Oguz, Pascal Denis, Simon Ostermann, Emmanuel Vincent, Natalia Skachkova, and Josef Van Genabith. (2024). "MMAR: Multilingual and Multimodal Anaphora Resolution in Instructional Videos." Findings of the Association for Computational Linguistics: EMNLP 2024.
Download Paper
Published in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024
Conversational explainable artificial intelligence (ConvXAI) systems based on large language models (LLMs) have garnered significant interest from the research community in natural language processing (NLP) and human-computer interaction (HCI). Such systems can provide answers to user questions about explanations in dialogues, have the potential to enhance users’ comprehension and offer more information about the decision-making and generation processes of LLMs. Currently available ConvXAI systems are based on intent recognition rather than free chat, as this has been found to be more precise and reliable in identifying users’ intentions. However, the recognition of intents still presents a challenge in the case of ConvXAI, since little training data exist and the domain is highly specific, as there is a broad range of XAI methods to map requests onto. In order to bridge this gap, we present CoXQL, the first dataset in the NLP domain for user intent recognition in ConvXAI, covering 31 intents, seven of which require filling multiple slots. Subsequently, we enhance an existing parsing approach by incorporating template validations, and conduct an evaluation of several LLMs on CoXQL using different parsing strategies. We conclude that the improved parsing approach (MP+) surpasses the performance of previous approaches. We also discover that intents with multiple slots remain highly challenging for LLMs.
Recommended citation: Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, and Sebastian Möller. (2024). "CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systems." Findings of the Association for Computational Linguistics: EMNLP 2024.
Download Paper
Published in Findings of the Association for Computational Linguistics: NAACL 2025, 2025
Recommended citation: Daniil Gurgurov, Rishu Kumar, and Simon Ostermann. (2025). "GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge." Findings of the Association for Computational Linguistics: NAACL 2025.
Download Paper
Published in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025
Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing (NLP). This study offers insights for improving cross-lingual NLP applications through the combination of parameter-efficient fine-tuning methods. We systematically explore strategies for enhancing cross-lingual transfer through the incorporation of language-specific and task-specific adapters and soft prompts. We present a detailed investigation of various combinations of these methods, exploring their efficiency across 16 languages, focusing on 10 mid- and low-resource languages. We further present to our knowledge the first use of soft prompts for language transfer, a technique we call soft language prompts. Our findings demonstrate that in contrast to claims of previous work, a combination of language and task adapters does not always work best; instead, combining a soft language prompt with a task adapter outperforms most configurations in many cases.
Recommended citation: Ivan Vykopal, Simon Ostermann, and Marian Simko. (2025). "Soft Language Prompts for Language Transfer." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).
Download Paper
Published in Proceedings of the 31st International Conference on Computational Linguistics, 2025
Natural language explanations (NLEs) are vital for elucidating the reasoning behind large language model (LLM) decisions. Many techniques have been developed to generate NLEs using LLMs. However, like humans, LLMs might not always produce optimal NLEs on first attempt. Inspired by human learning processes, we introduce Cross-Refine, which employs role modeling by deploying two LLMs as generator and critic, respectively. The generator outputs a first NLE and then refines this initial explanation using feedback and suggestions provided by the critic. Cross-Refine does not require any supervised training data or additional training. We validate Cross-Refine across three NLP tasks using three state-of-the-art open-source LLMs through automatic and human evaluation. We select Self-Refine (Madaan et al., 2023) as the baseline, which only utilizes self-feedback to refine the explanations. Our findings from automatic evaluation and a user study indicate that Cross-Refine outperforms Self-Refine. Meanwhile, Cross-Refine can perform effectively with less powerful LLMs, whereas Self-Refine only yields strong results with ChatGPT. Additionally, we conduct an ablation study to assess the importance of feedback and suggestions. Both of them play an important role in refining explanations. We further evaluate Cross-Refine on a bilingual dataset in English and German.
Recommended citation: Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, Sebastian Möller, and Vera Schmitt. (2025). "Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem." Proceedings of the 31st International Conference on Computational Linguistics.
Download Paper
Published in Proceedings of the 10th Workshop on Representation Learning for NLP (RepL4NLP-2025), 2025
In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer. Our source code is publicly available under https://github.com/Cora4NLP/multi-task-knowledge-transfer.
Recommended citation: Tatiana Anikina, Arne Binder, David Harbecke, Stalin Varanasi, Leonhard Hennig, Simon Ostermann, Sebastian Möller, and Josef Van Genabith. (2025). "Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution." Proceedings of the 10th Workshop on Representation Learning for NLP (RepL4NLP-2025).
Download Paper
Published in Accepted at European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD)., 2025
Prompt tuning is an efficient solution for training large language models (LLMs). However, current soft-prompt-based methods often sacrifice multi-task modularity, requiring the training process to be fully or partially repeated for each newly added task. While recent work on task vectors applied arithmetic operations on full model weights to achieve the desired multi-task performance, a similar approach for soft-prompts is still missing. To this end, we introduce Task Prompt Vectors, created by element-wise difference between weights of tuned soft-prompts and their random initialization. Experimental results on 12 NLU datasets show that task prompt vectors can be used in low-resource settings to effectively initialize prompt tuning on similar tasks. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 2 different language model architectures. This allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, we provide a competitive alternative to state-of-the-art baselines by arithmetic addition of task prompt vectors from multiple tasks.
Recommended citation: Robert Belanec, Simon Ostermann, Ivan Srba, and Maria Bielikova. (2025). "Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer." Accepted at European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD)..
Download Paper
Published in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), 2025
Low-resource languages (LRLs) face significant challenges in natural language processing (NLP) due to limited data. While current state-of-the-art large language models (LLMs) still struggle with LRLs, smaller multilingual models (mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of their capacity to low training data sizes. This study systematically investigates parameter-efficient adapter-based methods for adapting mLMs to LRLs, evaluating three architectures: Sequential Bottleneck, Invertible Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and structured knowledge from ConceptNet, we show that small adaptation datasets (e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains in intrinsic (masked language modeling) and extrinsic tasks (topic classification, sentiment analysis, and named entity recognition). We find that Sequential Bottleneck adapters excel in language modeling, while Invertible Bottleneck adapters slightly outperform other methods on downstream tasks due to better embedding alignment and larger parameter counts. Adapter-based methods match or outperform full fine-tuning while using far fewer parameters, and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3, GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves performance, pre-training data size remains the dominant factor, especially for languages with extensive pre-training coverage.The code for our experiments is available at https://github.com/d-gurgurov/Knowledge-Driven-Adaptation-LLMs.
Recommended citation: Daniil Gurgurov, Ivan Vykopal, Josef Van Genabith, and Simon Ostermann. (2025). "Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).
Download Paper
Published in Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), 2025
The rapid spread of online disinformation presents a global challenge, and machine learning has been widely explored as a potential solution. However, multilingual settings and low-resource languages are often neglected in this field. To address this gap, we conducted a shared task on multilingual claim retrieval at SemEval 2025, aimed at identifying fact-checked claims that match newly encountered claims expressed in social media posts across different languages. The task includes two subtracks: 1) a monolingual track, where social posts and claims are in the same language 2) a crosslingual track, where social posts and claims might be in different languages. A total of 179 participants registered for the task contributing to 52 test submissions. 23 out of 31 teams have submitted their system papers. In this paper, we report the best-performing systems as well as the most common and the most effective approaches across both subtracks. This shared task, along with its dataset and participating systems, provides valuable insights into multilingual claim retrieval and automated fact-checking, supporting future research in this field.
Recommended citation: Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podrouzek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. (2025). "SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval." Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025).
Download Paper
Published in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), 2025
Most state-of-the-art large language models (LLMs) are trained mainly on English data, limiting their effectiveness on non-English, especially low-resource, languages. This study investigates whether language adapters can facilitate cross-lingual transfer in English-centric LLMs. We train language adapters for 13 languages using Llama 2 (7B) and Llama 3.1 (8B) as base models, and evaluate their effectiveness on two downstream tasks (MLQA and SIB-200) using either task adapters or in-context learning. Our results reveal that language adapters improve performance for languages not seen during pretraining, but provide negligible benefit for seen languages. These findings highlight the limitations of language adapters as a general solution for multilingual adaptation in English-centric LLMs.
Recommended citation: Julian Schlenker, Jenny Kunz, Tatiana Anikina, Günter Neumann, and Simon Ostermann. (2025). "Only for the Unseen Languages, Say the Llamas: On the Efficacy of Language Adapters for Cross-lingual Transfer in English-centric LLMs." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).
Download Paper
Published in Findings of the Association for Computational Linguistics: ACL 2025, 2025
Counterfactual examples are widely used in natural language processing (NLP) as valuable data to improve models, and in explainable artificial intelligence (XAI) to understand model behavior. The automated generation of counterfactual examples remains a challenging task even for large language models (LLMs), despite their impressive performance on many tasks. In this paper, we first introduce ZeroCF, a faithful approach for leveraging important words derived from feature attribution methods to generate counterfactual examples in a zero-shot setting. Second, we present a new framework, FitCF, which further verifies aforementioned counterfactuals by label flip verification and then inserts them as demonstrations for few-shot prompting, outperforming three state-of-the-art baselines. Through ablation studies, we identify the importance of each of FitCF’s core components in improving the quality of counterfactuals, as assessed through flip rate, perplexity, and similarity measures. Furthermore, we show the effectiveness of LIME and Integrated Gradients as backbone attribution methods for FitCF and find that the number of demonstrations has the largest effect on performance. Finally, we reveal a strong correlation between the faithfulness of feature attribution scores and the quality of generated counterfactuals, which we hope will serve as an importantfinding for future research in this direction.
Recommended citation: Qianli Wang, Nils Feldhus, Simon Ostermann, Luis Felipe Villa-Arenas, Sebastian Möller, and Vera Schmitt. (2025). "FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation." Findings of the Association for Computational Linguistics: ACL 2025.
Download Paper
Published in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (Long Papers), 2025
Large Language Models (LLMs) are increasingly used to generate synthetic textual data for training smaller specialized models. However, a comparison of various generation strategies for low-resource language settings is lacking. While various prompting strategies have been proposed, such as demonstrations, label-based summaries, and self-revision, their comparative effectiveness remains unclear, especially for low-resource languages. In this paper, we systematically evaluate the performance of these generation strategies and their combinations across 11 typologically diverse languages, including several extremely low-resource ones. Using three NLP tasks and four open-source LLMs, we assess downstream model performance on generated versus gold-standard data. Our results show that strategic combinations of generation methods, particularly target-language demonstrations with LLM-based revisions, yield strong performance, narrowing the gap with real data to as little as 5\% in some settings. We also find that smart prompting techniques can reduce the advantage of larger LLMs, highlighting efficient generation strategies for synthetic data generation in low-resource scenarios with smaller models.
Recommended citation: Tatiana Anikina, Jan Cegin, Jakub Simko, and Simon Ostermann. (2025). "A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (Long Papers).
Download Paper
Published in Findings of the Association for Computational Linguistics: EMNLP 2025, 2025
In our era of widespread false information, human fact-checkers often face the challenge of duplicating efforts when verifying claims that may have already been addressed in other countries or languages. As false information transcends linguistic boundaries, the ability to automatically detect previously fact-checked claims across languages has become an increasingly important task. This paper presents the first comprehensive evaluation of large language models (LLMs) for multilingual previously fact-checked claim detection. We assess seven LLMs across 20 languages in both monolingual and cross-lingual settings. Our results show that while LLMs perform well for high-resource languages, they struggle with low-resource languages. Moreover, translating original texts into English proved to be beneficial for low-resource languages. These findings highlight the potential of LLMs for multilingual previously fact-checked claim detection and provide a foundation for further research on this promising application of LLMs.
Recommended citation: Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, Tatiana Anikina, Michal Gregor, and Marián Šimko. (2025). "Large Language Models for Multilingual Previously Fact-Checked Claim Detection." Findings of the Association for Computational Linguistics: EMNLP 2025.
Download Paper
Published in Findings of the Association for Computational Linguistics: EMNLP 2025, 2025
Conversational explainable artificial intelligence (ConvXAI) systems based on large language models (LLMs) have garnered considerable attention for their ability to enhance user comprehension through dialogue-based explanations. Current ConvXAI systems are often based on intent recognition to accurately identify the user’s desired intention and map it to an explainability method. While such methods offer great precision and reliability in discerning users’ underlying intentions for English, a significant challenge in the scarcity of training data persists, which impedes multilingual generalization. Besides, the support for free-form custom inputs, which are user-defined data distinct from pre-configured dataset instances, remains largely limited. To bridge these gaps, we first introduce MultiCoXQL, a multilingual extension of the CoXQL dataset spanning five typologically diverse languages, including one low-resource language. Subsequently, we propose a new parsing approach aimed at enhancing multilingual parsing performance, and evaluate three LLMs on MultiCoXQL using various parsing strategies. Furthermore, we present Compass, a new multilingual dataset designed for custom input extraction in ConvXAI systems, encompassing 11 intents across the same five languages as MultiCoXQL1. We conduct monolingual, cross-lingual, and multilingual evaluations on Compass, employing three LLMs of varying sizes alongside BERT-type models.
Recommended citation: Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, Fedor Splitt, Jiaao Li, Yoana Tsoneva, Sebastian Möller, and Vera Schmitt. (2025). "Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems." Findings of the Association for Computational Linguistics: EMNLP 2025.
Download Paper
Published in Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025), 2025
Recommended citation: Tanja Baeumel, Josef van Genabith, and Simon Ostermann. (2025). "Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms." Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025).
Download Paper
Published in Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2025
Autoregressive large language models (LLMs) exhibit impressive performance across various tasks but struggle with simple arithmetic, such as addition of two or more operands. We show that this struggle arises from LLMs’ use of a simple one-digit lookahead heuristic, which works fairly well (but not perfect) for two-operand addition but fails in multi-operand cases, where the carry-over logic is more complex. Our probing experiments and digit-wise accuracy evaluation show that LLMs fail precisely where a one-digit lookahead is insufficient to account for cascading carries. We analyze the impact of tokenization strategies on arithmetic performance and show that all investigated models, regardless of tokenization, are inherently limited in the addition of multiple operands due to their reliance on a one-digit lookahead heuristic. Our findings reveal fundamental limitations that prevent LLMs from generalizing to more complex numerical reasoning.
Recommended citation: Tanja Baeumel, Josef van Genabith, and Simon Ostermann. (2025). "The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs." Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP.
Download Paper
Published in The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Student Research Workshop), 2025
In this paper, we combine two-step knowledge distillation, structured pruning, truncation, and vocabulary trimming for extremely compressing multilingual encoder-only language models for low-resource languages. Our novel approach systematically combines existing techniques and takes them to the extreme, reducing layer depth, feed-forward hidden size, and intermediate layer embedding size to create significantly smaller monolingual models while retaining essential language-specific knowledge. We achieve compression rates of up to 92\% with only a marginal performance drop of 2-10\% in four downstream tasks, including sentiment analysis, topic classification, named entity recognition, and part-of-speech tagging, across three low-resource languages. Notably, the performance degradation correlates with the amount of language-specific data in the teacher model, with larger datasets resulting in smaller performance losses. Additionally, we conduct extensive ablation studies to identify best practices for multilingual model compression using these techniques.
Recommended citation: Daniil Gurgurov, Michal Gregor, Josef van Genabith, and Simon Ostermann. (2025). "On Multilingual Encoder Language Model Compression for Low-Resource Languages." The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Student Research Workshop).
Download Paper
Published in Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Findings), 2025
Large language models (LLMs) are increasingly used in everyday tools and applications, raising concerns about their potential influence on political views. While prior research has shown that LLMs often exhibit measurable political biases–frequently skewing toward liberal or progressive positions–key gaps remain. Most existing studies evaluate only a narrow set of models and languages, leaving open questions about the generalizability of political biases across architectures, scales, and multilingual settings. Moreover, few works examine whether these biases can be actively controlled. In this work, we address these gaps through a large-scale study of political orientation in modern open-source instruction-tuned LLMs. We evaluate seven models, including LLaMA-3.1, Qwen-3, and Aya-Expanse, across 14 languages using the Political Compass Test with 11 semantically equivalent paraphrases per statement to ensure robust measurement. Our results reveal that larger models consistently shift toward libertarian-left positions, with significant variations across languages and model families. To test the manipulability of political stances, we utilize a simple center-of-mass activation intervention technique and show that it reliably steers model responses toward alternative ideological positions across multiple languages. Our code is publicly available at https://github.com/d-gurgurov/Political-Ideologies-LLMs.
Recommended citation: Daniil Gurgurov, Katharina Trinley, Ivan Vykopal, Josef van Genabith, Simon Ostermann, and Roberto Zamparelli. (2025). "Multilingual Political Views of Large Language Models: Identification and Steering." Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Findings).
Download Paper
Published in Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Findings), 2025
Soft prompts have emerged as a powerful alternative to adapters in parameter-efficient fine-tuning (PEFT), enabling large language models (LLMs) to adapt to downstream tasks without architectural changes or parameter updates. While prior work has focused on stabilizing training via parameter interaction in small neural prompt encoders, their broader potential for transfer across languages remains unexplored. In this paper, we demonstrate that a prompt encoder can play a central role in improving performance on low-performing languages-those that achieve poor accuracy even under full-model fine-tuning. We introduce the Cross-Prompt Encoder (XPE), which combines a lightweight encoding architecture with multi-source training on typologically diverse languages - a design that enables the model to capture abstract and transferable patterns across languages. To complement XPE, we propose a Dual Soft Prompt mechanism that combines an encoder-based prompt with a directly trained standard soft prompt. This hybrid design proves especially effective for target languages that benefit from both broadly shared structure and language-specific alignment. Experiments on the SIB-200 benchmark reveal a consistent trade-off: XPE is most effective for low-performing languages, while hybrid variants offer broader adaptability across multilingual settings.
Recommended citation: Beso Mikaberidze, Teimuraz Saghinadze, Simon Ostermann, and Philipp Muller. (2025). "Cross-Prompt Encoder for Low-Performing Languages." Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Findings).
Download Paper
Published in Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Findings), 2025
While recent work has begun to uncover the internal strategies that Large Language Models (LLMs) employ for simple arithmetic tasks, a unified understanding of their underlying mechanisms is still lacking. We extend recent findings showing that LLMs represent numbers in a digit-wise manner and present evidence for the existence of digit-position-specific circuits that LLMs use to perform simple arithmetic tasks, i.e. modular subgroups of MLP neurons that operate independently on different digit positions (units, tens, hundreds). Notably, such circuits exist independently of model size and of tokenization strategy, i.e. both for models that encode longer numbers digit-by-digit and as one token. Using Feature Importance and Causal Interventions, we identify and validate the digit-position-specific circuits, revealing a compositional and interpretable structure underlying the solving of arithmetic problems in LLMs. Our interventions selectively alter the model’s prediction at targeted digit positions, demonstrating the causal role of digit-position circuits in solving arithmetic tasks.
Recommended citation: Tanja Baeumel, Daniil Gurgurov, Yusser al Ghussin, Josef van Genabith, and Simon Ostermann. (2025). "Modular Arithmetic: Language Models Solve Math Digit by Digit." Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Findings).
Download Paper
Published in Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Main), 2025
Large language models (LLMs) exhibit strong multilingual abilities, yet the neural mechanisms behind language-specific processing remain unclear. We analyze language-specific neurons in Llama-3.1-8B, Mistral-Nemo-12B, and Aya-Expanse-8B \& 32B across 21 typologically diverse languages, identifying neurons that control language behavior. Using the Language Activation Probability Entropy (LAPE) method, we show that these neurons cluster in deeper layers, with non-Latin scripts showing greater specialization. Related languages share overlapping neurons, reflecting internal representations of linguistic proximity. Through language arithmetics, i.e. systematic activation addition and multiplication, we steer models to deactivate unwanted languages and activate desired ones, outperforming simpler replacement approaches. These interventions effectively guide behavior across five multilingual tasks: language forcing, translation, QA, comprehension, and NLI. Manipulation is more successful for high-resource languages, while typological similarity improves effectiveness. We also demonstrate that cross-lingual neuron steering enhances downstream performance and reveal internal “fallback” mechanisms for language selection when neurons are progressively deactivated. Our code is made publicly available at https://github.com/d-gurgurov/Language-Neurons-Manipulation.
Recommended citation: Daniil Gurgurov, Katharina Trinley, Yusser Al Ghussin, Tanja Baeumel, Josef van Genabith, and Simon Ostermann. (2025). "Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation." Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Main).
Download Paper
Published in Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2026
Chat assistants increasingly integrate web search functionality, enabling them to retrieve and cite external sources. While this promises more reliable answers, it also raises the risk of amplifying misinformation from low-credibility sources. In this paper, we introduce a novel methodology for evaluating assistants’ web search behavior, focusing on source credibility and the groundedness of responses with respect to cited sources. Using 100 claims across five misinformation-prone topics, we assess GPT-4o, GPT-5, Perplexity, and Qwen Chat. Our findings reveal differences between the assistants, with Perplexity achieving the highest source credibility, whereas GPT-4o exhibits elevated citation of non-credibility sources on sensitive topics. This work provides the first systematic comparison of commonly used chat assistants for fact-checking behavior, offering a foundation for evaluating AI systems in high-stakes information environments.
Recommended citation: Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, and Marián Šimko. (2026). "Assessing Web Search Credibility and Response Groundedness in Chat Assistants." Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).
Download Paper
Published in Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), 2026
Recommended citation: Stelios Piperidis, Penny Labropoulou, Dimitrios Galanis, Khalid Choukri, Andrejs Vasiļjevs, Mitos Deligiannis, Katerina Gkirtzou, Dimitris Gkoumas, Athanasia Kolovou, Leon Voukoutis, Kanella Pouli, Maria Giagkou, Maria Gavriilidou, Katrin Marheinecke, Elena Leitner, Simon Ostermann, Stefania Raccioppa, Kossay Talmoudi, Victoria Arranz, Valérie Mapelli, Helene Mazo, Fernanda González Campo, Shi Yu, Aivars Bērziņş, Andis Lagzdiņş, and Georg Rehm. (2026). "Common European Language Data Space: Development, Current Status, and Future Perspectives." Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026).
Download Paper
Published in Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), 2026
Recommended citation: Christian Schuler, Raman Ahmad, Ānrán Wáng, Daniil Gurgurov, Timo Baumann, Simon Ostermann, and Josef van Genabith. (2026). "Dialectal Filtering: Synthesizing Kurdish Corpora for Low-Resource Varieties by Utilizing "Noise" in Large Textual Data." Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026).
Download Paper
Published in Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026), 2026
Quantization methods are widely used to accelerate inference and streamline the deployment of large language models (LLMs). Although quantization’s effects on various LLM capabilities have been extensively studied, one critical area remains underexplored: factual knowledge recall (FKR), the process by which LLMs access stored knowledge. To this end, we conduct comprehensive experiments using three common quantization techniques at distinct bit widths, in conjunction with interpretability-driven analyses on two tasks, knowledge memorization and latent multi-hop reasoning. We show that quantization typically results in information loss within LLMs, consequently diminishing their capacity for FKR. This effect is particularly amplified in smaller models within the same architectural families. However, models quantized at reduced bit precision do not consistently exhibit inferior performance and occasionally quantization may even enhance model FKR. We find that BitSandBytes demonstrates highest preservation of the original full-precision model’s FKR. Despite variability across models and methods, quantization causes modest performance degradation and remains an effective compression strategy.
Recommended citation: Qianli Wang, Mingyang Wang, Nils Feldhus, Simon Ostermann, Yuan Cao, Hinrich Schuetze, Sebastian Möller, and Vera Schmitt. (2026). "Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall." Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).
Download Paper
Published in Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026), 2026
Sparse autoencoders (SAEs) enable feature-level mechanistic interpretability and activation steering in large language models (LLMs), but SAE-based language control remains unreliable in multilingual settings: most SAEs are trained on English-only data, and steering layers are chosen heuristically. We address these limitations by advancing a principled, mechanistic account of multilingual language steering with SAEs. First, we show that training SAEs on multilingual data consistently strengthens cross-lingual representations and yields more reliable, quality-preserving language control across layers and model families. Second, we introduce an \textbackslashemph\a priori\ steering layer-selection rule based on the intersection of multilingual alignment and language separability, which predicts effective intervention depths without exhaustive layerwise search. We evaluate our approach on LLaMA-3.1-8B and Gemma-2-9B across machine translation and cross-lingual summarization (CrossSumm), using SpBLEU, ROUGE-L, COMET, and LaSE. Our results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality, providing a principled, predictive, representation-level account of multilingual SAE steering.
Recommended citation: Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel, Josef van Genabith, Patrick Schramowski, and Simon Ostermann. (2026). "Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection." Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).
Download Paper
Published in Findings of the Association for Computational Linguistics: ACL 2026, 2026
Recommended citation: Daniil Gurgurov, Yusser Al Ghussin, Tanja Baeumel, Cheng-Ting Chou, Patrick Schramowski, Marius Mosbach, Josef van Genabith, and Simon Ostermann. (2026). "CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark." Findings of the Association for Computational Linguistics: ACL 2026.
Download Paper
Published in Findings of the Association for Computational Linguistics: ACL 2026, 2026
Recommended citation: Cennet Oguz, Yasser Hamidullah, Josef van Genabith, and Simon Ostermann. (2026). "DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning." Findings of the Association for Computational Linguistics: ACL 2026.
Download Paper
Published in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026
Recommended citation: Dan Shi, Zhuowen Han, Simon Ostermann, Renren Jin, Josef van Genabith, and Deyi Xiong. (2026). "Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Download Paper
Published in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026
Recommended citation: Simon Ostermann, Daniil Gurgurov, Tanja Baeumel, Michael A. Hedderich, Sebastian Lapuschkin, Wojciech Samek, and Vera Schmitt. (2026). "From Weights to Activations: Is Steering the Next Frontier of Adaptation?." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Download Paper
Published in Accepted at the 9th BlackboxNLP Workshop, 2026
This study examines the compositionality of steering vectors for language and behavioral control in large language models. Focusing on language, jailbreak, and conciseness, we investigate whether additive, training-free composition of attribute steering vectors can preserve the intended steering effect of each attribute, across four instruction-tuned models from two model families and two size scales. We find that single-attribute steering is reliable for all three attributes, but only within an appropriate combination of intervention layer and steering strength, with abstract behaviors (jailbreak, conciseness) favoring middle layers and language favoring earlier layers. We show that additive composition of two attribute vectors succeeds in steering both attributes simultaneously when each is injected at its own best-performing layer, and that this partially extends to three simultaneously composed attributes, addressing an inconsistency left open by prior work on training-free composition. We further analyze the geometric properties of these steering vectors, finding that they are approximately orthogonal in the residual stream, consistent with their compositional behavior.
Recommended citation: Hyun Gu Kang, Daniil Gurgurov, Tanja Baeumel, Josef van Genabith, and Simon Ostermann. (2026). "Compositional Multilingual and Behavioral Attribute Steering." Accepted at the 9th BlackboxNLP Workshop.
Published in Accepted at the 9th BlackboxNLP Workshop Special Track: Reproducibility and Reliability in Interpretability Analyses, 2026
Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model’s outputs. Since human evaluation is costly, automated simulatability replaces human explainees with LLM simulators, as proposed in ConSim (Poché et al., 2025) for large-scale experiments. We qualitatively replicate and extend ConSim’s ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs, and identify two limitations. First, when class names are meaningful, simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations. Second, class anonymization can reward explanations for leaking the hidden label mapping, a limitation we expose with a new classes-as-concepts baseline. These results are consistent with a shortcut hypothesis: in the tested settings, simulator predictions mainly rely on task priors, while explanations produce small changes. We derive recommendations for more robust automated simulatability evaluations.
Recommended citation: Antonin Poché, Fanny Jourdan, Nils Feldhus, Qianli Wang, Jing Yang, Simon Ostermann, Nicholas Asher, Philippe Muller, and Vera Schmitt. (2026). "Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations." Accepted at the 9th BlackboxNLP Workshop Special Track: Reproducibility and Reliability in Interpretability Analyses.
Published in Accepted at the 9th BlackboxNLP Workshop, 2026
Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the $\textbackslashtextit\representational\$ level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model’s output script while largely maintaining semantic content. The vector generalizes asymmetrically to writing systems unseen during construction, flipping non-Latin output to Latin reliably, but mapping Latin output into varied non-Latin scripts. At the $\textbackslashtextit\mechanistic\$ level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the network. Collectively, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.
Recommended citation: Daniil Gurgurov, Alan Saji, Katharina Trinley, Josef van Genabith, and Simon Ostermann. (2026). "The Latin Substrate: How Language Models Represent and Mediate Script Choice." Accepted at the 9th BlackboxNLP Workshop.
Published in Accepted at the Findings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026
Recommended citation: Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov, Josef van Genabith, and Simon Ostermann. (2026). "Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs." Accepted at the Findings of the 2026 Conference on Empirical Methods in Natural Language Processing.
Download Paper
Published in Accepted at the Findings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026
Recommended citation: Qianli Wang, Nils Feldhus, Pepa Atanasova, Fedor Splitt, Simon Ostermann, Sebastian Möller, and Vera Schmitt. (2026). "Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanations." Accepted at the Findings of the 2026 Conference on Empirical Methods in Natural Language Processing.
Download Paper
Published in Accepted at the Findings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026
Recommended citation: Yilong Wang, Qianli Wang, Bohao Chu, Yihong Liu, Jing Yang, and Simon Ostermann. (2026). "Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization." Accepted at the Findings of the 2026 Conference on Empirical Methods in Natural Language Processing.
Download Paper
Published in Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (Main), 2026
Use Google Scholar for full citation
Recommended citation: Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet, Josef van Genabith, and Simon Ostermann. (2026). "FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing." Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (Main).
Published in Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (Main), 2026
Recommended citation: Tanja Baeumel, Josef van Genabith, and Simon Ostermann. (2026). "When Tokenization is Secretly Output Supervision." Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (Main).
Download Paper
Published in Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (Main), 2026
Recommended citation: Jan Cegin, Daniil Gurgurov, Yusser Al Ghussin, and Simon Ostermann. (2026). "Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation." Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (Main).
Download Paper
Published:
Published:
Published:
Published:
Published:
Published:
Published:
Published:
Seminar, Saarland University, 2022
Software Project, Saarland University, 2023
Seminar, Saarland University, 2023
Seminar, Saarland University, 2024
Seminar, Saarland University, 2024
Seminar, Saarland University, 2024
Software Project, Saarland University, 2024
Seminar, Saarland University, 2025
Lecture, Saarland University, 2025
Seminar, Saarland University, 2026