Breaking Boundaries: Investigating the Effects of
Model Editing on Cross-linguistic Performance

Somnath Banerjee † Avik Halder † Rajarshi Mandal †11footnotemark: 1 Sayan Layek †
Ian Soboroff ‡ Rima Hazra ∓ Animesh Mukherjee †
†Indian Institute of Technology Kharagpur, India
‡National Institute of Standards and Technology, USA
∓Singapore University of Technology and Design, Singapore
{som.iitkgpcse,avik08,rajarshimandal,sayanlayek2002}@kgpian.iitkgp.ac.in
{rima_hazra}@sutd.edu.sg
These authors contributed equally to this work.
Abstract

The integration of pretrained language models (PLMs) like BERT and GPT has revolutionized NLP, particularly for English, but it has also created linguistic imbalances. This paper strategically identifies the need for linguistic equity by examining several knowledge editing techniques in multilingual contexts. We evaluate the performance of models such as Mistral, TowerInstruct, OpenHathi, Tamil-Llama, and Kan-Llama across languages including English, German, French, Italian, Spanish, Hindi, Tamil, and Kannada. Our research identifies significant discrepancies in normal and merged models concerning cross-lingual consistency. We employ strategies like ‘each language for itself’ (ELFI) and ‘each language for others’ (ELFO) to stress-test these models. Our findings demonstrate the potential for LLMs to overcome linguistic barriers, laying the groundwork for future research in achieving linguistic inclusivity in AI technologies 111Repository: https://github.com/NeuralSentinel/BreakingBoundaries.

\newmdenv

[ topline=false, bottomline=false, skipabove=skipbelow=leftline=true, rightline=false, linecolor=red, linewidth=4pt, innertopmargin=10pt, innerbottommargin=2pt, innerrightmargin=2pt, innerleftmargin=2pt, backgroundcolor=gray!10, roundcorner=10pt ]stylishframe

Breaking Boundaries: Investigating the Effects of
Model Editing on Cross-linguistic Performance


Somnath Banerjee † Avik Halder ††thanks: Theseauthorscontributedequallytothiswork.†superscript††thanks: Theseauthorscontributedequallytothiswork.†{}^{\dagger}\lx@make@thanks{Theseauthorscontributedequallytothiswork.}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT Theseauthorscontributedequallytothiswork. Rajarshi Mandal †11footnotemark: 1 Sayan Layek † Ian Soboroff ‡ Rima Hazra ∓ Animesh Mukherjee † †Indian Institute of Technology Kharagpur, India ‡National Institute of Standards and Technology, USA ∓Singapore University of Technology and Design, Singapore {som.iitkgpcse,avik08,rajarshimandal,sayanlayek2002}@kgpian.iitkgp.ac.in {rima_hazra}@sutd.edu.sg


1 Introduction

The advent of model editing techniques, as explored by Sinitsin et al. (2020) and De Cao et al. (2021), introduces an innovative avenue for refining LLM responses to specific inputs, marking a significant stride toward addressing these challenges. However, applying these techniques within a cross-lingual framework introduces a set of unique obstacles Qi et al. (2023); Xu et al. (2023), necessitating the development of inventive strategies to ensure the effective and equitable operation of multilingual PLMs across the globe’s linguistic tapestry Wang et al. (2023).

Refer to caption
Figure 1: Edited knowledge conflict across various languages for TowerInstruct.

Building upon this foundation, recent scholarly endeavors have increasingly focused on the nuances of knowledge editing within LLMs, particularly within monolingual contexts Hazra et al. (2024); Banerjee et al. (2024). This body of work has catalyzed significant advancements in the creation and refinement of models tailored for multilingual usage, such as Bloom Workshop et al. (2023), ChatGPT222https://chat.openai.com/, LLaMA Touvron et al. (2023). Yet, the broader implications of implementing these knowledge edits across diverse linguistic landscapes remain an area ripe for exploration.
The exploration of cross-lingual consistency is paramount for several compelling reasons. Primarily, it challenges the conventional wisdom that knowledge is inextricably linked to the form of language, advocating instead for a model’s comprehension to remain steadfast across linguistic divisions. For instance, a model’s recognition of “Dent Island Light, located in: Belgium” (Post Edit) (see Figure 1) should be consistent, irrespective of the language employed. Such consistency is crucial for ensuring a uniform user experience across different languages, thereby democratizing access to information and technology.
In pursuit of these objectives, our study conducts a comprehensive examination of the efficacy with which multilingual models transfer knowledge across eight distinct languages tackling the largely unexplored challenges associated with maintaining knowledge consistency and fostering language diversity within the sphere of knowledge editing for LLMs. Inspired by the earlier research Beniwal et al. (2024) in this domain, our investigation explores various strategies for disseminating factual knowledge from linguistically rich to relatively resource-scarce languages, employing the “each language for itself” (ELFI) and “each language for others” (ELFO) principle as a foundational premise Das et al. (2022). Our findings illustrate the intricate dynamics and untapped potential of knowledge transfer in multilingual contexts, underscoring the complexities, challenges, and opportunities that lie ahead in our quest to achieve linguistic equity in the age of artificial intelligence. Our contributions include:

  • •

    We conduct extensive experiments on model editing across eight distinct languages (English (En), German (De), French (Fr), Italian (It) and Spanish (Es from Romance and Germanic scripts, whereas Hindi (Hi), Tamil (Ta) and Kannada (Kn)) from Indic scripts, using two approaches: “each language for itself” (ELFI) and “each language for others” (ELFO). Our observations specifically focused on the performance of decoder-only models from a multilingual perspective.

  • •

    Second, this study explores decoder-only models, including state-of-the-art systems such as Mistral, TowerInstruct, OpenHathi, Tamil-Llama and Kan-Llama (all 7B models). We utilize these models in conjunction with well-known editing methods, ROME and MEMIT, marking a significant advancement in the field.

  • •

    Third, to the best of our knowledge, this paper is the first to evaluate that while model merging enhances capabilities, it still falls short in maintaining cross-lingual consistency post editing.

  • •

    Our comprehensive error analysis reveals significant areas where linguistic discrepancies across different languages lead to varied interpretations and meanings. This study delves into how such differences can cause models to generate incorrect responses post edit.

2 Related work

Targeted parameter editing specifically focuses on identifying and modifying particular components within a model to integrate new information. This approach was illustrated in Dai et al. (2022) through the identification and adjustment of ‘knowledge neurons’ in transformer models, allowing for the reflection of new data. Meng and colleagues furthered this concept with the Rank-One Model Editing (ROME) Meng et al. (2022) technique, which aimed to update key neural network weights, refreshing factual knowledge in LLMs. Despite their effectiveness for singular updates, these techniques faced difficulties with multiple simultaneous updates. MEMIT Meng et al. (2023), a successor of ROME, addressed this by enabling the concurrent modification of various knowledge points, a breakthrough supported by subsequent studies like Hase et al. (2023); Yao et al. (2023).
Multilingual knowledge editing Research in knowledge editing has predominantly focused on English. However, a handful of studies have ventured into the multilingual domain, translating English-based prompts and object pairs into various languages. Initiatives like X-FACTR Jiang et al. (2020) and M-LAMA Kassner et al. (2021) have identified substantial knowledge disparities across languages, attributed to differences in training data volume. These discrepancies are particularly pronounced for languages other than English and some European languages, where probing accuracies often fall below 10%. GeoMLAMA Yin et al. (2022), took a unique approach by examining region-specific commonsense knowledge, revealing that a country’s native language may not always be the most effective for accessing specific national knowledge.
Our research takes a systematic step in analyzing the cross-lingual consistency of factual knowledge within multilingual LLMs, assessing how consistently an LLM provides the same answer to the same question across different languages. Earlier works, such as those by Wang et al. (2023); Beniwal et al. (2024), initiated this exploration with mBERT, uncovering low consistencies. We expand on this by including various LLMs finetuned on specific languages.

3 Task overview

Model editing: Given a language model θp⁢r⁢esubscript𝜃𝑝𝑟𝑒\theta_{pre}italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT and an edit descriptor <k⁢n𝑘𝑛knitalic_k italic_n, an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT, ao⁢l⁢dsubscript𝑎𝑜𝑙𝑑a_{old}italic_a start_POSTSUBSCRIPT italic_o italic_l italic_d end_POSTSUBSCRIPT>, the model editing technique will create an edited model θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT. So, for an input prompt k⁢n𝑘𝑛knitalic_k italic_n, θp⁢r⁢esubscript𝜃𝑝𝑟𝑒\theta_{pre}italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT has the old prediction ao⁢l⁢dsubscript𝑎𝑜𝑙𝑑a_{old}italic_a start_POSTSUBSCRIPT italic_o italic_l italic_d end_POSTSUBSCRIPT and after editing θp⁢r⁢esubscript𝜃𝑝𝑟𝑒\theta_{pre}italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT, the edited model θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT has updated prediction an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT without influencing model behaviour on other samples. Thus, given the edit input k⁢n𝑘𝑛knitalic_k italic_n, θp⁢r⁢esubscript𝜃𝑝𝑟𝑒\theta_{pre}italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT does not produce an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT; it is θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT that is designed to produce the output an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT.

θe⁢d⁢i⁢t⁢(k⁢n)={an⁢e⁢wif ⁢k⁢n∈I⁢(k⁢n,an⁢e⁢w)θp⁢r⁢e⁢(k⁢n)if ⁢k⁢n∈O⁢(k⁢n,an⁢e⁢w)subscript𝜃𝑒𝑑𝑖𝑡𝑘𝑛casessubscript𝑎𝑛𝑒𝑤if 𝑘𝑛𝐼𝑘𝑛subscript𝑎𝑛𝑒𝑤subscript𝜃𝑝𝑟𝑒𝑘𝑛if 𝑘𝑛𝑂𝑘𝑛subscript𝑎𝑛𝑒𝑤{\theta_{edit}}(kn)=\begin{cases}a_{new}&\text{if }kn\in I(kn,a_{new})\\ {\theta_{pre}}(kn)&\text{if }kn\in O(kn,a_{new})\end{cases}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT ( italic_k italic_n ) = { start_ROW start_CELL italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT end_CELL start_CELL if italic_k italic_n ∈ italic_I ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT ( italic_k italic_n ) end_CELL start_CELL if italic_k italic_n ∈ italic_O ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ) end_CELL end_ROW (1)

The scope of consideration, I⁢(k⁢n,an⁢e⁢w)𝐼𝑘𝑛subscript𝑎𝑛𝑒𝑤I(kn,a_{new})italic_I ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ), includes k⁢n𝑘𝑛knitalic_k italic_n and similar versions of it. This means it covers the original input and any rephrased versions of it that still relate to the same topic. For example, if k⁢n𝑘𝑛knitalic_k italic_n is a question, this scope includes different ways of asking the same question. However, the excluded scope, O⁢(k⁢n,an⁢e⁢w)𝑂𝑘𝑛subscript𝑎𝑛𝑒𝑤O(kn,a_{new})italic_O ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ), refers to inputs that are not related to the edit case provided. So, it leaves out any inputs that do not have anything to do with k⁢n𝑘𝑛knitalic_k italic_n or its related versions. Along with the updated information, the edited model should follow the four properties: (i) reliability – θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT, produces the correct response for the specific edit scenario represented by (k⁢n𝑘𝑛knitalic_k italic_n, an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT), (ii) generalization – the edited model θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT must uniformly apply edits to both the designated edit case (k⁢n𝑘𝑛knitalic_k italic_n, an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT) and its semantically equivalent variations, guaranteeing a consistent output, an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT, across all rephrased iterations of k⁢n𝑘𝑛knitalic_k italic_n, (iii) locality – θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT should not alter the output for examples outside its intended scope (O(k⁢n𝑘𝑛knitalic_k italic_n, an⁢e⁢wsubscript𝑎𝑛𝑒𝑤a_{new}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT)), and (iv) portability – evaluates the capacity of edited model θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT for robust generalization, assessed through questions designed to test the edited model’s reasoning with updated knowledge.

Multilingual knowledge editing: Given a set of languages ℒℒ\mathcal{L}caligraphic_L, we consider a language l∈ℒ𝑙ℒl\in\mathcal{L}italic_l ∈ caligraphic_L to edit the model θp⁢r⁢esubscript𝜃𝑝𝑟𝑒\theta_{pre}italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT and obtain θe⁢d⁢i⁢tlsuperscriptsubscript𝜃𝑒𝑑𝑖𝑡𝑙\theta_{edit}^{l}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l end_POSTSUPERSCRIPT. We then test the edited model θe⁢d⁢i⁢tlsuperscriptsubscript𝜃𝑒𝑑𝑖𝑡𝑙\theta_{edit}^{l}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l end_POSTSUPERSCRIPT with all the languages in ℒℒ\mathcal{L}caligraphic_L. In the equations below, s𝑠sitalic_s is the source language, and t𝑡titalic_t is the target language. The conditions are as follows: if k⁢ns𝑘subscript𝑛𝑠kn_{s}italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT is in the inclusion scope I⁢(k⁢n,an⁢e⁢w)𝐼𝑘𝑛subscript𝑎𝑛𝑒𝑤I(kn,a_{new})italic_I ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ), the model should output an⁢e⁢wssuperscriptsubscript𝑎𝑛𝑒𝑤𝑠a_{new}^{s}italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_s end_POSTSUPERSCRIPT. Otherwise, if k⁢ns𝑘subscript𝑛𝑠kn_{s}italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT is in the exclusion scope O⁢(k⁢n,an⁢e⁢w)𝑂𝑘𝑛subscript𝑎𝑛𝑒𝑤O(kn,a_{new})italic_O ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ), the model should output θp⁢r⁢e⁢(k⁢ns)subscript𝜃𝑝𝑟𝑒𝑘subscript𝑛𝑠\theta_{pre}(kn_{s})italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT ( italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ). For the target language, similar conditions apply with transformations 𝒯tsuperscript𝒯𝑡\mathcal{T}^{t}caligraphic_T start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT.

θe⁢d⁢i⁢t⁢(k⁢ns)={an⁢e⁢wsif ⁢k⁢ns∈I⁢(k⁢n,an⁢e⁢w)θp⁢r⁢e⁢(k⁢ns)if ⁢k⁢ns∈O⁢(k⁢n,an⁢e⁢w)subscript𝜃𝑒𝑑𝑖𝑡𝑘subscript𝑛𝑠casessuperscriptsubscript𝑎𝑛𝑒𝑤𝑠if 𝑘subscript𝑛𝑠𝐼𝑘𝑛subscript𝑎𝑛𝑒𝑤subscript𝜃𝑝𝑟𝑒𝑘subscript𝑛𝑠if 𝑘subscript𝑛𝑠𝑂𝑘𝑛subscript𝑎𝑛𝑒𝑤{\theta_{edit}}(kn_{s})=\begin{cases}a_{new}^{s}&\text{if }kn_{s}\in I(kn,a_{% new})\\ \theta_{pre}(kn_{s})&\text{if }kn_{s}\in O(kn,a_{new})\end{cases}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT ( italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) = { start_ROW start_CELL italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_s end_POSTSUPERSCRIPT end_CELL start_CELL if italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ∈ italic_I ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ) end_CELL end_ROW start_ROW start_CELL italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT ( italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) end_CELL start_CELL if italic_k italic_n start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ∈ italic_O ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ) end_CELL end_ROW (2)
θe⁢d⁢i⁢t⁢(k⁢nt)={𝒯t⁢(an⁢e⁢ws)if ⁢k⁢nt∈𝒯t⁢(I⁢(k⁢n,an⁢e⁢w))θp⁢r⁢e⁢(k⁢nt)if ⁢k⁢nt∉𝒯t⁢(O⁢(k⁢n,an⁢e⁢w))subscript𝜃𝑒𝑑𝑖𝑡𝑘subscript𝑛𝑡casessuperscript𝒯𝑡superscriptsubscript𝑎𝑛𝑒𝑤𝑠if 𝑘subscript𝑛𝑡superscript𝒯𝑡𝐼𝑘𝑛subscript𝑎𝑛𝑒𝑤subscript𝜃𝑝𝑟𝑒𝑘subscript𝑛𝑡if 𝑘subscript𝑛𝑡superscript𝒯𝑡𝑂𝑘𝑛subscript𝑎𝑛𝑒𝑤\theta_{edit}(kn_{t})=\begin{cases}\mathcal{T}^{t}(a_{new}^{s})&\text{if }kn_{% t}\in\mathcal{T}^{t}(I(kn,a_{new}))\\ \theta_{pre}(kn_{t})&\text{if }kn_{t}\notin\mathcal{T}^{t}(O(kn,a_{new}))\end{cases}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT ( italic_k italic_n start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) = { start_ROW start_CELL caligraphic_T start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_s end_POSTSUPERSCRIPT ) end_CELL start_CELL if italic_k italic_n start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ caligraphic_T start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ( italic_I ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ) ) end_CELL end_ROW start_ROW start_CELL italic_θ start_POSTSUBSCRIPT italic_p italic_r italic_e end_POSTSUBSCRIPT ( italic_k italic_n start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_CELL start_CELL if italic_k italic_n start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∉ caligraphic_T start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ( italic_O ( italic_k italic_n , italic_a start_POSTSUBSCRIPT italic_n italic_e italic_w end_POSTSUBSCRIPT ) ) end_CELL end_ROW (3)

𝒯t(.)\mathcal{T}^{t}(.)caligraphic_T start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ( . ) transforms the target output of the source language to the target language with the same meaning. Therefore, after editing the model in one language, such as English, the effect of the edit should be reflected in other languages as well. This ensures that the specific edit is consistent across all languages, regardless of the language in which the edit was made.
Model merging: In the specific case of Indic languages – Hindi, Tamil and Kannada – we have specialized LLMs for each unlike in the case of Western languages where the models we have used are known to be pretrained on all those languages. We investigate if the three LLMs for the Indic languages could be further unified to obtain a more powerful model θm⁢e⁢r⁢g⁢e⁢dsubscript𝜃𝑚𝑒𝑟𝑔𝑒𝑑\theta_{merged}italic_θ start_POSTSUBSCRIPT italic_m italic_e italic_r italic_g italic_e italic_d end_POSTSUBSCRIPT, which dynamically harnesses the specialized linguistic capabilities of each constituent models. This involves extracting language-specific unique task vectors from instruction-tuned models, i.e., θb⁢a⁢s⁢e−H⁢i⁢n⁢d⁢i→v→H⁢i⁢n⁢d⁢i→subscript𝜃𝑏𝑎𝑠𝑒𝐻𝑖𝑛𝑑𝑖subscript→𝑣𝐻𝑖𝑛𝑑𝑖\theta_{base-Hindi}\rightarrow\vec{v}_{Hindi}italic_θ start_POSTSUBSCRIPT italic_b italic_a italic_s italic_e - italic_H italic_i italic_n italic_d italic_i end_POSTSUBSCRIPT → over→ start_ARG italic_v end_ARG start_POSTSUBSCRIPT italic_H italic_i italic_n italic_d italic_i end_POSTSUBSCRIPT, θb⁢a⁢s⁢e−T⁢a⁢m⁢i⁢l→v→T⁢a⁢m⁢i⁢l→subscript𝜃𝑏𝑎𝑠𝑒𝑇𝑎𝑚𝑖𝑙subscript→𝑣𝑇𝑎𝑚𝑖𝑙\theta_{base-Tamil}\rightarrow\vec{v}_{Tamil}italic_θ start_POSTSUBSCRIPT italic_b italic_a italic_s italic_e - italic_T italic_a italic_m italic_i italic_l end_POSTSUBSCRIPT → over→ start_ARG italic_v end_ARG start_POSTSUBSCRIPT italic_T italic_a italic_m italic_i italic_l end_POSTSUBSCRIPT, and θb⁢a⁢s⁢e−K⁢a⁢n⁢n⁢a⁢d⁢a→v→K⁢a⁢n⁢n⁢a⁢d⁢a→subscript𝜃𝑏𝑎𝑠𝑒𝐾𝑎𝑛𝑛𝑎𝑑𝑎subscript→𝑣𝐾𝑎𝑛𝑛𝑎𝑑𝑎\theta_{base-Kannada}\rightarrow\vec{v}_{Kannada}italic_θ start_POSTSUBSCRIPT italic_b italic_a italic_s italic_e - italic_K italic_a italic_n italic_n italic_a italic_d italic_a end_POSTSUBSCRIPT → over→ start_ARG italic_v end_ARG start_POSTSUBSCRIPT italic_K italic_a italic_n italic_n italic_a italic_d italic_a end_POSTSUBSCRIPT for each respective language. These vectors are integrated using a TIES Yadav et al. (2023) merging technique to synthesize θm⁢e⁢r⁢g⁢e⁢dsubscript𝜃𝑚𝑒𝑟𝑔𝑒𝑑\theta_{merged}italic_θ start_POSTSUBSCRIPT italic_m italic_e italic_r italic_g italic_e italic_d end_POSTSUBSCRIPT. Subsequently, θm⁢e⁢r⁢g⁢e⁢dsubscript𝜃𝑚𝑒𝑟𝑔𝑒𝑑\theta_{merged}italic_θ start_POSTSUBSCRIPT italic_m italic_e italic_r italic_g italic_e italic_d end_POSTSUBSCRIPT is edited in the same process as above to obtain θe⁢d⁢i⁢tsubscript𝜃𝑒𝑑𝑖𝑡\theta_{edit}italic_θ start_POSTSUBSCRIPT italic_e italic_d italic_i italic_t end_POSTSUBSCRIPT each time adjusting its output specifically for inputs associated with the defined task and the language.

4 Dataset

For our experiments, we use the popular CounterFact Meng et al. (2022) and ZsRE Levy et al. (2017) datasets. We uniformly sample ∼similar-to\sim∼ 550 edit instances from each dataset. Each edit instance in these datasets includes the actual edit case, the reliability prompt, the generalization instances, the locality prompt and its answer, portability and its answer. Further we use google translator 333https://translate.google.com/ to translate each edit instance into seven other languages – German (De), French (Fr), Italian (It), Spanish (Es), Hindi (Hi), Tamil (Ta) and Kannada (Kn). In both the datasets, the actual portability prompt is an interrogative sentence (i.e., in the form of question). However, when the question gets translated to other languages, the translated question becomes different from actual question format. For example, when the actual portability prompt in English “To which language family does the official language of Sastamala belong?” is translated to French the new prompt becomes “À quelle langue la famille appartient la langue officielle de Sastamala?”. However when this is back-translated to English the prompt means “Which family language does the official language of Sastamala belong to?” which is not the same as the original English prompt. We therefore employed GPT-4444https://openai.com/research/gpt-4, version: gpt-4-0125-preview to convert question in the interrogative sentence into a task of sentence completion. Subsequently we translate this sentence completion form to other languages to obtain the corresponding portability prompt.
Note to the choice of languages: The Western languages that we choose are based on their cultural, economic and academic significance Lobachev (2008)555https://preply.com/en/blog/most-important-languages/ and cover the Romance and the Germanic families. In addition, we include three Indic languages that have far lesser resources compared to their Western counterparts.

5 Experimental setup

5.1 Selection of LLMs

We use the following multilingual LLMs for our experiments.
Mistral-7B-Instruct-v0.2666https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2: The model was developed by Jiang et al. (2023) and supports multilinguality777https://encord.com/blog/mistral-large-explained/. It is designed around the causal language modeling framework. We shall refer to this model as Mistral.
TowerInstruct-7B-v0.2888https://huggingface.co/Unbabel/TowerInstruct-7B-v0.2: This model Alves et al. (2024) has been developed on top of LLaMA2 Touvron et al. (2023) architecture and supports multilinguality including English, German, French, Spanish, Chinese, Portuguese, Italian, Russian, Korean, and Dutch. We shall refer to this model as TowerInstruct.
OpenHathi-7B-Hi-v0.1-Base999https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base: The model is designed to optimize multilingual interactions with a special focus on Indian languages. It uses a transformer-based architecture similar to GPT-3 but introduces hybrid partitioned attention to efficiently manage computational resources and enhance responsiveness across languages like Hindi, Tamil, and Bengali. We shall refer to this model as OpenHathi.
Tamil-llama-7b-base-v0.1101010https://huggingface.co/abhinand/tamil-llama-7b-base-v0.1: This is a sophisticated model developed specifically for bilingual tasks in Tamil and English, leveraging a 7 billion parameter causal language modeling framework. We shall refer to this model as Tamil-Llama.
Kan-LLaMA-7B-SFT111111https://huggingface.co/Tensoic/Kan-Llama-7B-SFT-v0.5: This model is tailored for efficient Kannada text processing with an expanded 49,420-token vocabulary, enhancing its language handling capabilities. Pre-trained on 600 million Kannada tokens from the CulturaX dataset, it employs a low-rank adaptation technique to minimize computational costs while preserving the model’s integrity. We shall refer to this model as Kan-Llama.

Counterfact Models TowerInstruct Mistral OpenHathi Tamil-Llama Kan-Llama
Languages Metrics ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT
De Rel 0.83/0.96 0.73/0.83 0.83/0.96 0.73/0.87 - - - - - -
Gen 0.27/0.31 0.19/0.22 0.28/0.31 0.19/0.22 - - - - - -
Loc 0.22/0.23 0.19/0.22 0.21/0.23 0.24/0.27 - - - - - -
Port 0.01/0.01 0.01/0.01 0.03/0.04 0.04/0.06 - - - - - -
Es Rel 0.82/0.92 0.70/0.80 0.81/0.91 0.78/0.86 - - - - - -
Gen 0.33/0.37 0.23/0.27 0.28/0.32 0.22/0.27 - - - - - -
Loc 0.21/0.22 0.19/0.19 0.25/0.27 0.27/0.29 - - - - - -
Port 0.00/0.00 0.00/0.00 0.03/0.03 0.03/0.04 - - - - - -
It Rel 0.87/0.93 0.74/0.78 0.86/0.91 0.80/0.88 - - - - - -
Gen 0.35/0.38 0.25/0.26 0.28/0.30 0.24/0.27 - - - - - -
Loc 0.18/0.19 0.20/0.20 0.26/0.27 0.27/0.28 - - - - - -
Port 0.02/0.02 0.02/0.03 0.02/0.03 0.03/0.03 - - - - - -
Fr Rel 0.83/0.90 0.65/0.72 0.83/0.89 0.79/0.85 - - - - - -
Gen 0.31/0.33 0.22/0.24 0.29/0.30 0.24/0.25 - - - - - -
Loc 0.21/0.22 0.17/0.19 0.20/0.22 0.24/0.25 - - - - - -
Port 0.00/0.01 0.00/0.00 0.03/0.03 0.03/0.03 - - - - - -
Hi Rel - - - - 0.02/0.02 0.45/0.60 - - - -
Gen - - - - 0.00/0.00 0.26/0.33 - - - -
Loc - - - - 0.31/0.35 0.02/0.03 - - - -
Port - - - - 0.01/0.01 0.01/0.01 - - - -
Ta Rel - - - - - - 0.12/0.15 0.48/0.59 - -
Gen - - - - - - 0.03/0.04 0.21/0.25 - -
Loc - - - - - - 0.01/0.01 0.01/0.01 - -
Port - - - - - - 0.01/0.01 0.01/0.01 - -
Kn Rel - - - - - - - - 0.21/0.26 0.14/0.18
Gen - - - - - - - - 0.07/0.08 0.04/0.05
Loc - - - - - - - - 0.03/0.04 0.02/0.03
Port - - - - - - - - 0.00/0.00 0.00/0.01
Table 1: Comparison of reliability, generalization, locality, and portability scores across various language models, evaluated using the CounterFact dataset under same language edit and inference settings. The highest score is presented in bold whereas the second highest is underlined.

5.2 Editing methods

We use ROME (Rank-One Model Editing) Meng et al. (2022) and MEMIT (Mass Editing Memory in a Transformer) Meng et al. (2023) which are the state-of-the-art editing schemes and particularly suitable for multilingual settings.
Rank-One Model Editing (ROME): This method specifically alters the weights in the initial feed-forward layers of a pretrained model. It identifies factual associations through causal interventions, enabling precise and effective modifications.
Mass Editing Memory in a Transformer (MEMIT): MEMIT advances ROME, by extending its capabilities. While ROME applied a rank-one modification to the MLP weights of a single layer to embed a memory directly into the model, MEMIT enhances this approach by adjusting the MLP weights across multiple critical layers to incorporate numerous memories.

5.3 Evaluation metric

We evaluate the generated output from the edited models on four properties using two different metrics as follows.
Exact match: In this evaluation method, we systematically determine the accuracy of outputs generated through our model by checking for the presence of the ground truth within these outputs. The ground truth represents the exact, correct response expected from the model for a given input. An output is considered accurate (a ‘correct generation’) if it includes the ground truth, indicating the model’s ability to replicate the desired response accurately. Conversely, if the ground truth is not present in the output, it is classified as inaccurate (‘incorrect generation’). This binary evaluation approach provides a clear and direct measure of the model’s performance, focusing on its ability to produce outputs that are not only relevant but also correct as per the predefined standards.
Partial match: In case of partial match, the Levenshtein ratio is utilized as an alternative measure of textual similarity. This ratio is calculated by dividing the Levenshtein distance Levenshtein (1965), by the maximum length of either the ground truth text or the generated text. A critical threshold is set at an 80% Levenshtein ratio; outputs that do not contain the ground truth as a substring but surpass this ratio are deemed accurate. This approach allows for a nuanced assessment of the generated text’s correctness, accommodating minor deviations or errors that do not significantly detract from the overall meaning or accuracy of the generated output.

6 Results

ZsRE Models TowerInstruct Mistral OpenHathi Tamil-Llama Kan-Llama
Languages Properties ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT
De Rel 0.48/0.59 0.25/0.30 0.51/0.62 0.38/0.47 - - - - - -
Gen 0.33/0.39 0.11/0.12 0.35/0.45 0.18/0.24 - - - - - -
Loc 0.00/0.01 0.00/0.01 0.01/0.02 0.01/0.03 - - - - - -
Port 0.02/0.02 0.00/0.00 0.08/0.10 0.02/0.04 - - - - - -
Es Rel 0.44/0.59 0.24/0.34 0.49/0.61 0.37/0.49 - - - - - -
Gen 0.30/0.40 0.16/0.20 0.35/0.45 0.22/0.29 - - - - - -
Loc 0.00/0.01 0.01/0.02 0.01/0.01 0.02/0.02 - - - - - -
Port 0.02/0.02 0.01/0.02 0.03/0.07 0.03/0.04 - - - - - -
It Rel 0.54/0.62 0.25/0.29 0.58/0.65 0.42/0.50 - - - - - -
Gen 0.35/0.43 0.16/0.20 0.42/0.48 0.25/0.31 - - - - - -
Loc 0.00/0.00 0.00/0.01 0.00/0.02 0.01/0.02 - - - - - -
Port 0.01/0.02 0.02/0.03 0.07/0.08 0.01/0.03 - - - - - -
Fr Rel 0.51/0.59 0.27/0.35 0.52/0.63 0.40/0.50 - - - - - -
Gen 0.28/0.35 0.14/0.17 0.40/0.50 0.19/0.27 - - - - - -
Loc 0.00/0.01 0.00/0.02 0.01/0.02 0.01/0.02 - - - - - -
Port 0.03/0.05 0.03/0.03 0.06/0.09 0.04/0.06 - - - - - -
Hi Rel - - - - 0.03/0.06 0.20/0.33 - - - -
Gen - - - - 0.01/0.04 0.19/0.28 - - - -
Loc - - - - 0.01/0.01 0.00/0.01 - - - -
Port - - - - 0.00/0.00 0.03/0.03 - - - -
Ta Rel - - - - - - 0.06/0.08 0.16/0.21 - -
Gen - - - - - - 0.03/0.04 0.10/0.14 - -
Loc - - - - - - 0.00/0.00 0.00/0.00 - -
Port - - - - - - 0.00/0.00 0.01/0.01 - -
Kn Rel - - - - - - - - 0.16/0.21 0.05/0.07
Gen - - - - - - - - 0.08/0.17 0.05/0.05
Loc - - - - - - - - 0.00/0.00 0.00/0.00
Port - - - - - - - - 0.00/0.01 0.00/0.00
Table 2: Comparison of reliability, generalization, locality, and portability scores across various language models, evaluated for the ZsRE dataset under same language edit and inference settings. Each cell shows results in the format: exact match/partial match. The highest score presented in bold whereas second highest presented in underlined.
CounterFact Models TowerInstruct Mistral OpenHathi Tamil-Llama Kan-Llama
Languages Properties ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT
De Rel 0.48/0.53 0.40/0.46 0.50/0.56 0.54/0.61 - - - - - -
Gen 0.25/0.27 0.13/0.17 0.23/0.27 0.22/0.23 - - - - - -
Loc 0.20/0.21 0.19/0.22 0.23/0.25 0.26/0.28 - - - - - -
Port 0.00/0.00 0.00/0.00 0.03/0.03 0.03/0.04 - - - - - -
Es Rel 0.51/0.56 0.40/0.48 0.57/0.62 0.56/0.60 - - - - - -
Gen 0.26/0.29 0.18/0.22 0.25/0.29 0.21/0.26 - - - - - -
Loc 0.22/0.24 0.17/0.17 0.24/0.27 0.25/0.27 - - - - - -
Port 0.00/0.00 0.00/0.00 0.03/0.03 0.03/0.04 - - - - - -
It Rel 0.45/0.50 0.35/0.40 0.47/0.58 0.44/0.49 - - - - - -
Gen 0.23/0.27 0.19/0.20 0.25/0.35 0.21/0.23 - - - - - -
Loc 0.20/0.21 0.20/0.20 0.24/0.36 0.28/0.29 - - - - - -
Port 0.01/0.02 0.01/0.02 0.03/0.11 0.04/0.04 - - - - - -
Fr Rel 0.50/0.53 0.45/0.49 0.49/0.55 0.51/0.59 - - - - - -
Gen 0.28/0.31 0.19/0.22 0.28/0.31 0.26/0.27 - - - - - -
Loc 0.23/0.23 0.19/0.21 0.20/0.36 0.25/0.26 - - - - - -
Port 0.01/0.01 0.01/0.01 0.01/0.12 0.03/0.04 - - - - - -
Hi Rel - - - - 0.56/0.66 0.02/0.03 - - - -
Gen - - - - 0.27/0.34 0.03/0.03 - - - -
Loc - - - - 0.26/0.31 0.03/0.03 - - - -
Port - - - - 0.02/0.02 0.00/0.01 - - - -
Ta Rel - - - - - - 0.00/0.00 0.01/0.01 - -
Gen - - - - - - 0.00/0.00 0.00/0.00 - -
Loc - - - - - - 0.01/0.01 0.01/0.02 - -
Port - - - - - - 0.00/0.00 0.00/0.00 - -
Kn Rel - - - - - - - - 0.01/0.01 0.00/0.00
Gen - - - - - - - - 0.00/0.01 0.00/0.00
Loc - - - - - - - - 0.02/0.02 0.03/0.03
Port - - - - - - - - 0.00/0.00 0.00/0.00
Table 3: Comparison of reliability, generalization, locality, and portability scores across various language models, evaluated for the CounterFact dataset under English edit and target language inference settings. Each cell shows results in the format: exact match/partial match.
ZsRE Models TowerInstruct Mistral OpenHathi Tamil-Llama Kan-Llama
Languages Properties ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT
De Rel 0.24/0.28 0.10/0.14 0.34/0.45 0.14/0.18 - - - - - -
Gen 0.18/0.23 0.12/0.14 0.26/0.35 0.14/0.16 - - - - - -
Loc 0.00/0.01 0.00/0.02 0.01/0.02 0.01/0.03 - - - - - -
Port 0.02/0.02 0.02/0.02 0.06/0.07 0.02/0.03 - - - - - -
Es Rel 0.24/0.29 0.12/0.14 0.39/0.48 0.19/0.26 - - - - - -
Gen 0.18/0.25 0.09/0.11 0.33/0.41 0.14/0.21 - - - - - -
Loc 0.00/0.01 0.01/0.02 0.01/0.02 0.02/0.02 - - - - - -
Port 0.02/0.03 0.01/0.01 0.04/0.06 0.04/0.05 - - - - - -
It Rel 0.24/0.29 0.12/0.14 0.31/0.34 0.23/0.27 - - - - - -
Gen 0.17/0.22 0.11/0.13 0.26/0.32 0.18/0.21 - - - - - -
Loc 0.00/0.00 0.00/0.01 0.00/0.02 0.01/0.02 - - - - - -
Port 0.01/0.02 0.02/0.02 0.07/0.08 0.01/0.01 - - - - - -
Fr Rel 0.22/0.26 0.12/0.17 0.36/0.44 0.23/0.28 - - - - - -
Gen 0.15/0.21 0.08/0.10 0.29/0.33 0.16/0.21 - - - - - -
Loc 0.00/0.01 0.00/0.02 0.01/0.03 0.01/0.02 - - - - - -
Port 0.02/0.02 0.02/0.02 0.06/0.09 0.04/0.05 - - - - - -
Hi Rel - - - - 0.03/0.03 0.03/0.06 - - - -
Gen - - - - 0.03/0.03 0.04/0.08 - - - -
Loc - - - - 0.00/0.00 0.00/0.01 - - - -
Port - - - - 0.00/0.00 0.01/0.01 - - - -
Ta Rel - - - - - - 0.00/0.00 0.01/0.01 - -
gen - - - - - - 0.00/0.00 0.00/0.00 - -
Loc - - - - - - 0.00/0.00 0.00/0.00 - -
Port - - - - - - 0.00/0.00 0.00/0.00 - -
Kn Rel - - - - - - - - 0.00/0.01 0.00/0.01
gen - - - - - - - - 0.00/0.00 0.00/0.00
Loc - - - - - - - - 0.00/0.00 0.00/0.00
Port - - - - - - - - 0.00/0.00 0.00/0.00
Table 4: Comparison of reliability, generalization, locality, and portability scores across various language models, evaluated for the ZsRE dataset under English edit and target language inference settings. Each cell shows results in the format: exact match/partial match.
Dataset CounterFact ZsRE
Inferencing language En Hi Ta Kn En Hi Ta Kn
Editing language Properties ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT ROME MEMIT
En Rel 0.73/0.75 0.95/0.95 0.00/0.00 0.01/0.01 0.00/0.00 0.01/0.01 0.00/0.01 0.00/0.01 0.29/0.33 0.59/0.59 0.01/0.02 0.02/0.02 0.00/0.00 0.00/0.00 0.00/0.02 0.00/0.00
Gen 0.35/0.35 0.64/0.64 0.01/0.01 0.02/0.02 0.01/0.01 0.01/0.02 0.00/0.01 0.00/0.01 0.29/0.31 0.52/0.54 0.01/0.02 0.00/0.00 0.01/0.01 0.00/0.00 0.00/0.03 0.00/0.00
Loc 0.33/0.33 0.27/0.27 0.01/0.01 0.01/0.01 0.02/0.02 0.03/0.03 0.11/0.11 0.12/0.12 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.01/0.01 0.00/0.04 0.01/0.02 0.02/0.04
Port 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.03/0.04 0.02/0.04 0.00/0.01 0.00/0.00 0.00/0.01 0.00/0.00 0.00/0.01 0.00/0.00
Hi Rel 0.00/0.01 0.01/0.01 0.01/0.03 0.07/0.09 0.00/0.00 0.01/0.01 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.01/0.03 0.05/0.05 0.00/0.00 0.00/0.00 0.00/0.02 0.00/0.01
Gen 0.00/0.00 0.01/0.01 0.02/0.03 0.03/0.04 0.00/0.00 0.01/0.01 0.00/0.01 0.00/0.01 0.00/0.00 0.01/0.01 0.01/0.03 0.02/0.03 0.01/0.02 0.01/0.02 0.00/0.03 0.00/0.02
Loc 0.35/0.35 0.35/0.36 0.01/0.01 0.01/0.01 0.03/0.03 0.03/0.03 0.12/0.12 0.13/0.13 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.01/0.01 0.01/0.01 0.01/0.01 0.01/0.01
Port 0.00/0.00 0.00/0.00 0.01/0.01 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.07/0.08 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.01 0.00/0.01 0.00/0.01 0.00/0.01
Ta Rel 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.01 0.01/0.01 0.00/0.01 0.00/0.01 0.00/0.00 0.01/0.01 0.00/0.00 0.01/0.01 0.00/0.02 0.01/0.03 0.00/0.01 0.00/0.01
Gen 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.01/0.01 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.00 0.01/0.01 0.00/0.00 0.01/0.01 0.01/0.01 0.02/0.03 0.00/0.02 0.00/0.02
Loc 0.36/0.36 0.33/0.34 0.01/0.01 0.02/0.02 0.02/0.02 0.02/0.02 0.11/0.11 0.11/0.11 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.01/0.03 0.01/0.02 0.01/0.02 0.01/0.02
Port 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.01/0.01 0.00/0.01 0.00/0.01 0.00/0.01
Kn Rel 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.00 0.00/0.02 0.00/0.00 0.03/0.03 0.00/0.03
Gen 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.00 0.01/0.03 0.01/0.02 0.01/0.03 0.00/0.04
Loc 0.35/0.35 0.34/0.34 0.01/0.01 0.02/0.02 0.03/0.03 0.03/0.03 0.12/0.12 0.12/0.12 0.00/0.00 0.00/0.00 0.01/0.01 0.00/0.00 0.00/0.01 0.00/0.00 0.01/0.01 0.00/0.00
Port 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.01 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.00 0.00/0.01 0.00/0.00 0.00/0.01 0.00/0.00 0.00/0.01 0.00/0.01
Table 5: Comparison of reliability, generalization, locality, and portability scores across the merge model for the three Indic languages, evaluated using the CounterFact and ZsRE dataset under each language for itself and others. Each cell shows results in the format: exact match/partial match. The highest score presented in bold whereas second highest presented in underlined. The highest score presented in bold whereas second highest presented in underlined.

6.1 Self edit - self inference perspective

In this setup we perform the edit in a particular language (say German) and obtain the generated output from the model in the same language (i.e., German itself).
CounterFact dataset: In our evaluations of the model performance for the CounterFact dataset, we observe marked variations across different languages and metrics in Table 1, illustrating significant challenges in multilingual adaptability and contextual understanding. For instance, German language tests show that models like TowerInstruct and Mistral achieve good reliability scores (ROME at 0.83 and MEMIT at 0.73 for TowerInstruct; the same scores are at 0.83 and 0.73 respectively for Mistral). These scores illustrate good model performance in understanding the contextual nuances of German. However, generalization and locality score are less impressive (TowerInstruct at 0.27 and 0.22 on ROME for generalization and locality respectively), indicating difficulties in applying the learned information across broader contexts and different locales within the German language. Similar patterns are observed in Spanish and Italian. In Spanish, TowerInstruct reaches a reliability score of 0.82 for ROME and 0.70 for MEMIT; for Mistral the reliability scores are 0.81 for ROME and 0.78 for MEMIT, suggesting decent grasp of Spanish contexts. However, the generalization scores remain below 0.35 for ROME and locality scores do not exceed 0.29 for MEMIT for any model. Despite TowerInstruct showing a relatively high reliability in Italian with a ROME at 0.87 and MEMIT at 0.74, the generalization and locality scores remain low (highest being 0.35 on ROME and 0.28 on MEMIT for Mistral). In case of the three Indic languages the discrepancies become even more pronounced. OpenHathi, for example, shows a drastic drop in Hindi, with a ROME reliability of just 0.02 and a MEMIT of 0.45, indicating almost no comprehension of the language nuances. Tamil-Llama and Kan-Llama also display low scores across all properties. The highest reliability achieved is 0.21 for ROME for Kan-Llama and 0.48 for MEMIT in case of Tamil-Llama, which highlights the limitations in these language models. Portability scores are consistently low across all languages, models, and metrics, demonstrating a significant gap in model training as it fails to effectively account for diverse linguistic structures and cultural contexts.

Id Category Examples Possible solution
1 Lexical ambiguity
English: ‘Fair’ can mean a carnival, treating someone right, or having light skin and/or hair
French: ‘Livre’ can refer to a book or to the weight measure pound.
Context-aware models
2 Syntactic ambiguity
English: “Visiting relatives can be boring.” (Ambiguous: Visiting them, or the relatives who visit, can be boring.)
German: “Er sah den Mann mit dem Fernglas.” (He saw the man with the binoculars. Ambiguous: Who has the binoculars?)
Italian: “Ho visto l’uomo con il binocolo.” (I saw the man with the binocular. Ambiguous similar to German.)
Better parsing
3 Semantic ambiguity
French: “Mexx, ça a commencé en” (Mexx, that was started in. Ambiguous: started means founded or
started in a particular region)
Spanish: “Spike Hughes se origina de” (Spike Hughes originates from. Ambiguous: originates from a place or
from a particular family)
Incorporation of additional semantic cues
4 Cultural ambiguity
English: “Arrow of Time/The Cycle of Time” (Is an album of Peter Michael Hamel. But it could also mean the flow of time)
French: “Ce n’est pas ma tasse de thé.” (It’s not my cup of tea. Ambiguous without understanding the idiom.)
Italian: “In bocca al lupo.” (In the wolf’s mouth, means good luck. Could be confusing without cultural context.)
Deeper multi-cultural context
5 Translation errors
English: “In which country’s capital city would you most likely
hear Faithless’ original language spoken?” translated into French and back to English becomes “In which
country’s capital would you most likely hear the original language of the original spoken”
Reinterpretation of the translation in target language
6 NER errors
English: “The Little Match Girl” could be a literary fairy tale.
Spanish: ‘Rio’ can mean a river or refer to the city Rio de Janeiro.
Integration of knowledge graphs
7 Idioms
German: “Der Blick von unten” (Literally: Seeing things from a low physical position. Meaning: Considering
a situation from a marginalized or disadvantaged perspective.)
Maintain exception lists
8 Phonetic/orthographic errors
English: ‘Their’ vs. ‘There’ vs. ‘They’re’
Spanish: ‘Vino’ (came) vs. ‘Vino’ (wine)
Context-sensitive correction of word forms
9 Morphological errors
German: The misuse of gender-specific articles "der" (masculine), "die" (feminine), "das" (neuter) can lead to confusion
Italian: Confusion between "mangiato" (eaten) and "mangiando" (eating) can change the temporal context of a sentence.
Integration of specialised morphological rules
10 Pragmatic errors
French: Using ‘tu’ (informal you) instead of ‘vous’ (formal or plural you) in a formal context can be seen as rude or too casual.
Understanding cultural norms
Table 6: A systematic categorization of errors encountered in the process of multilingual knowledge editing, spanning lexical, syntactic, semantic, cultural, and contextual ambiguities, among others. Each category is illustrated with examples from English, French, German, Italian, and Spanish, showcasing the multifaceted challenges of maintaining consistency and accuracy in knowledge representation across languages.

ZsRE dataset: In case of ZsRE dataset (see Table 2) German shows moderate performance in reliability with scores like 0.48 on ROME and 0.25 on MEMIT for TowerInstruct. The generalization (0.33 for ROME) and locality scores (∼0similar-toabsent0\sim 0∼ 0) are also very poor. These results indicate substantial deficiencies in capturing language-specific details and generalizing learned information. Spanish fares slightly better in reliability, achieving up to 0.49 on ROME with TowerInstruct and Mistral, but like German, faces challenges in generalization and locality, with the best generality reaching only 0.35 and locality remaining near zero. Italian (It) generally scores higher in reliability, particularly with Mistral reaching 0.58 on ROME, though it too struggles with generality and locality. French exhibits a similar trend, with reliability scores reaching up to 0.52 for ROME with Mistral and both generalization and locality scores remaining low. Performance markedly drops for the three Indic languages. For instance, Hindi’s highest reliability is just 0.03 for ROME, while Tamil and Kannada only achieve maximum reliability scores of 0.06 and 0.16 respectively for ROME. Across all languages, portability scores are low, reflecting limited adaptability and the challenge of transferring learned capabilities from one linguistic context to another.

6.2 English edit - self inference perspective

In this setup we perform the edit in a English and obtain the generated output from the model in other languages (e.g., German, Italian etc.).

• Inconsistent model performance Models like TowerInstruct and Mistral exhibit varied performance across languages, excelling in context-specific reliability but faltering in generalization and locality. • Struggle with Indic languages: Discrepancies are particularly pronounced in Indic languages, where models struggle significantly more compared to European languages, underscoring a need for better linguistic diversity in model training. • Cross-Linguistic weaknesses: Cross-lingual edits highlight critical deficiencies, with dramatic performance drops when models attempt to process and understand edits across linguistic boundaries. • Merged model outcomes: Merging models do not improve their capabilities. Neither the reliability nor the locality or generalization scores improve in any of the two datasets due to merging. \pgfmathresultptKey observations  \pgfmathresultptKey observations  

CounterFact dataset: In German, the reliability scores for models such as TowerInstruct and Mistral suggest moderate effectiveness, with ROME around 0.48 and MEMIT around 0.40 (see Table 3). However, their generalization and locality scores reveal limitations in the models’ ability to generalize and localize content effectively with scores not exceeding 0.25 and 0.26 respectively. For Spanish, there is a noticeable improvement in reliability, with ROME scores for Mistral reaching 0.57, and a slight improvement in generalization and locality metrics compared to German. Italian and French show similar trends, with reliability scores peaking at 0.47 for Mistral in Italian and 0.49 in French; the generalization and locality scores are still lower. For Tamil and Kannada the reliability are exceptionally low. In fact, in case of Tamil this score is 0 for ROME and 0.01 for MEMIT. Comparatively for Hindi the reliability scores are quite good with 0.56 for ROME. However the portability and generalization scores are again very poor.
ZsRE dataset: For languages such as German and Spanish, the models display moderate reliability with Mistral, achieving ROME scores up to 0.34 and 0.39 respectively, and MEMIT scores of 0.14 and 0.19 respectively (see Table 4). However, the scores significantly drop for locality and portability, showing that while the models can identify relevant relationships, they struggle to generalize and adapt to the specific linguistic nuances of these languages. The trends are similar in Italian and French, where reliability scores are moderate while locality and generalization scores are poor. Further, for the Indic languages, the score are exceedingly low for all the properties indicating the stark gap in performance highly resource scarce languages. .

6.3 Merged model perspective

The Table 5 presents performance metrics for the merged model, showcasing how it handles edits across various languages, with columns representing the inferencing language and rows indicating the respective editing language. When both editing and inferencing are done in English the reliability scores for the CounterFact dataset are quite high with ROME and MEMIT respectively reaching 0.73 and 0.95. However, the performance sharply declines when editing in English and inferencing is in Hindi, Tamil, and Kannada with scores nearly zero, highlighting a stark limitation in the model’s cross-lingual capabilities. This trend is consistently observed across both datasets. Editing in languages like Hindi, Tamil, or Kannada results in uniformly poor outcomes across all four properties regardless of the inferencing language, which indicates profound inadequacies in the model’s ability to generalize and adapt across linguistic barriers. This pattern emphasizes the critical need for advancing multilingual model adaptability. The current findings suggest that while the model operates effectively within the confines of the same linguistic environment, its performance deteriorates dramatically across linguistic boundaries, especially from lesser-resourced languages. Such insights advocate for a significant enhancement in training approaches, aiming to foster robust multilingual support and ensure that models are truly multilingual in functionality, proficiently managing edits and inferencing across a diverse linguistic landscape.

7 Error analysis

In Table 6 we show the different types of linguistic errors encountered during the translation and editing process. The errors are categorised based on the different types of ambiguities and sheds light on how future models should strengthened by carefully harnessing techniques to tackle these errors.

8 Conclusion

In this study, we investigated the impact of knowledge editing across different languages based on the CounterFact and ZsRE datasets along with their translations. Our extensive experiments employing a variety of knowledge editing techniques on an array of multilingual LLMs resulted in various crucial observations. We discovered that variations in language-specific model architecture significantly affect the success of knowledge edits, that current editing methods often fail to seamlessly transfer alterations from one language to another, and that modifications made in one language might unexpectedly alter model behavior in another language. This study lays the groundwork for future innovations that could lead to more sophisticated and linguistically inclusive AI technologies.

References

Appendix A Hyperparameters

We adopt all essential parameter values from the ROME and MEMIT study for all the LLMs. The details of these hyperparameters are provided in Table 7.

Hyperparameter values
layers [5]
fact_token subject_last
v_num_grad_steps 25
v_lr 5e-1
v_loss_layer 31
v_weight_decay 1e-3
clamp_norm_factor 4
kl_factor 0.0625
mom2_adjustment false
context_template_length_params [[5, 10], [10, 10]]
rewrite_module_tmp model.layers.{}.mlp.down_proj
layer_module_tmp model.layers.{}
mlp_module_tmp model.layers.{}.mlp
attn_module_tmp model.layers.{}.self_attn
ln_f_module model.norm
lm_head_module lm_head
model_parallel true
Table 7: Hyperparameter values (most of the default values extend from ROME and MEMIT setup).

Appendix B Exact vs partial match

We showcase plot correlations in Figures 2 and 3.

Refer to caption
Figure 2: Each metric on the x𝑥xitalic_x-axis is represented by two bars: the left bar indicates an exact match, while the right bar indicates a partial match. For each bar, the divisions along the y𝑦yitalic_y-axis reflect the average values of the metric, aggregated across Romance and Germanic languages evaluated. These subdivisions are color-coded to denote the editing language, as specified in the legend.
Refer to caption
Figure 3: Each metric on the x𝑥xitalic_x-axis is represented by two bars: the left bar indicates an exact match, while the right bar indicates a partial match. For each bar, the divisions along the y𝑦yitalic_y-axis reflect the average values of the metric, aggregated across all Indic languages evaluated. These subdivisions are color-coded to denote the editing language, as specified in the legend.

Appendix C Romance and Germanic languages

C.1 Language perspective

C.1.1 CounterFact

In case of CounterFact dataset, significant disparities are observed in edited model performance across different languages. Edits done with En and tested on En consistently showed high reliability scores across all models, with Mistral achieving nearly perfect reliability at 0.994 and TowerInstruct at 0.996 (for ROME). However, performances while testing with De, It, Fr, and Es were notably lower, particularly in generalisation (in between ∼similar-to\sim∼0.21-0.28 for Mistral) and locality (0.20-0.28 for Mistral) metrics, indicating challenges in generalization and nuanced information processing in non-English contexts. The portability scores were modest across the board, underscoring a pronounced need for enhanced multilingual model adaptability.

When the edit is conducted with De and tested on De reliability scores for TowerInstruct (0.828) and Mistral (0.834) (for ROME) are reasonably high indicating strong contextual understanding. However, testing with other languages like It, Fr, and Es exhibit lower scores, reflecting challenges in language-specific processing.

After editing the model with It the edited model achieved the highest reliability score with TowerInstruct for test language It (0.871) (for ROME). However, the reliability scores for other test languages were lower, with En at 0.535, De at 0.398, Fr at 0.490, and Es at 0.488, reflecting the challenge of extending training efficiencies beyond Italian. The highest portability score was seen in It with Mistral and TowerInstruct at 0.095 (for ROME), the scores were significantly lower in other languages.

In case of edit with Fr, test language Fr achieved the highest scores (0.832), with TowerInstruct where it reached 0.454, compared to model’s performance in other languages like En (0.519), De (0.417), It (0.509), and Es (0.511). This high score in Fr for TowerInstruct, however, suggests that certain models can still effectively align with training data even in non-primary languages. In case generality and locality, the scores were universally lower across all models and languages, indicating a struggle in generalizing the Fr editing. Locality scores also pointed to difficulties in identifying language-specific nuances, with TowerInstruct showing a modestly better understanding in It (0.189) and Fr (0.214), yet still remaining low.

After editing with Es, En (0.555) consistently demonstrated superior reliability score for TowerInstruct, compared to other languages such as De (0.391) and It (0.451) (excluding Es). However, Es exhibited notably high reliability scores, with TowerInstruct achieving 0.822 and Mistral 0.812, indicating these models’ effective adaptation to Spanish linguistic features. Generality and locality metrics, which measure a model’s ability to generalize training and identify language-specific information, respectively, showed universally lower scores across all languages, highlighting challenges in cross-lingual applicability.

C.1.2 ZsRE

After editing with En language, the reliability score for Mistral model in En was remarkably high at 0.929. However, this contrasts sharply with its performance in other languages such as De (0.344) and It (0.312), suggesting a significant drop in model effectiveness when transitioning from  En. Similarly, the TowerInstruct model showed a strong performance when the test langauage was En with a relevance score of 0.875, yet scores in other languages like De (0.236) and Fr (0.221) were markedly lower, highlighting the challenges in maintaining model performance across linguistic boundaries (for ROME). In case of generalization and locality, the scores also emphasize the disparity. While Mistral displayed a good generality in Eng (0.812), its scores in languages such as De and It were only around 0.260. This trend of decreased performance is echoed in the locality scores, where Mistral exhibited almost no ability to identify language-specific nuances in It and Fr. TowerInstruct’s portability score for En was 0.097, which, although not very high, still outperforms its De and Fr counterparts, suggesting a somewhat better but still limited ability to adapt training across languages (for ROME).

After editing with De, the TowerInstruct model exhibited significant variations in reliability scores, achieving its highest in De (0.480) but only 0.157 in En, indicating a substantial challenge in adapting to De compared to other languages. Similarly, Mistral displayed relatively better relevance in De at 0.513, but this still fell short compared to its performance in It (0.257), suggesting a consistent trend of models performing better in Romance languages. Further examination of generalization and locality metrics highlights these disparities even more. For instance, generalization scores for Mistral in De stood at 0.349, yet locality scores were nearly zero across the board, showing a significant deficiency in capturing language-specific details. Portability scores also reflect limited adaptability, with Mistral scoring only 0.079 for De compared to a slightly better performance in It (0.066), underscoring the need for model training approaches that better address and bridge these linguistic gaps to enhance overall performance and applicability across diverse linguistic datasets (for ROME).

After editing with It, TowerInstruct model exhibited a disparity in reliability scores, achieving a high value of 0.537 in It but only 0.185 in De, underscoring a significant challenge in adapting to De compared to other Romance languages. Similarly, Mistral demonstrated better reliability in It (0.575), further indicating that models tend to align more effectively with training data in certain languages over others. In terms of generality and locality, the scores further emphasize these challenges.

After editing with Fr, the TowerInstruct demonstrated a stronger performance in Fr with a reliability score of 0.507 and a generality score of 0.281, compared to its performance in Es (Rel: 0.138, Gen: 0.113) and It (Rel: 0.197, Gen: 0.167). This indicates a more robust alignment with Fr linguistic features. On the other hand, Mistral also exhibited its highest reliability in Fr (0.517) but struggled in De (0.298) and It (0.272), further underscoring the varying model efficiencies across languages. These findings highlight significant challenges in model training, where improvements are needed to enhance language-specific understanding and adaptability, ensuring that models perform consistently well across a diverse linguistic spectrum.

After editing with Es, TowerInstruct achieved a high reliability score of 0.443 for Es, significantly surpassing its scores in other languages such as En (0.232) and De (0.148). This trend suggests a stronger model alignment with the linguistic properties of Es. In generality, TowerInstruct highlights better performance in Es with a score of 0.305, contrasted with lower scores in It (0.202) and Fr (0.182). The locality scores were generally low across all languages.

Datasets/ Languages Score Mistral TowerInstruct
En De It Fr Es En De It Fr Es
CounterFact En Rel 0.994/0.994 0.498/0.560 0.469/0.578 0.487/0.548 0.571/0.617 0.996/0.996 0.482/0.529 0.455/0.500 0.498/0.527 0.511/0.562
Gen 0.512/0.529 0.233/0.269 0.246/0.346 0.279/0.305 0.252/0.294 0.522/0.538 0.245/0.273 0.231/0.267 0.280/0.309 0.256/0.291
Loc 0.327/0.338 0.227/0.250 0.240/0.358 0.200/0.362 0.244/0.265 0.307/0.315 0.196/0.207 0.204/0.209 0.225/0.235 0.224/0.238
Port 0.133/0.144 0.029/0.033 0.027/0.111 0.013/0.119 0.027/0.035 0.005/0.013 0.000/0.004 0.011/0.018 0.005/0.005 0.002/0.004
De Rel 0.558/0.591 0.834/0.961 0.471/0.506 0.423/0.471 0.446/0.500 0.589/0.614 0.828/0.959 0.431/0.489 0.439/0.481 0.429/0.497
Gen 0.355/0.394 0.284/0.313 0.266/0.303 0.255/0.286 0.245/0.282 0.322/0.345 0.271/0.314 0.211/0.246 0.224/0.246 0.224/0.255
Loc 0.365/0.376 0.208/0.228 0.251/0.264 0.193/0.207 0.263/0.280 0.287/0.292 0.222/0.232 0.212/0.216 0.214/0.224 0.211/0.224
Port 0.114/0.133 0.029/0.039 0.025/0.027 0.023/0.023 0.033/0.037 0.004/0.014 0.008/0.008 0.004/0.006 0.006/0.012 0.000/0.002
It Rel 0.541/0.578 0.422/0.477 0.860/0.914 0.502/0.542 0.519/0.582 0.535/0.564 0.398/0.450 0.871/0.932 0.490/0.535 0.488/0.556
Gen 0.319/0.346 0.202/0.218 0.278/0.296 0.235/0.239 0.235/0.267 0.330/0.349 0.226/0.253 0.346/0.376 0.263/0.290 0.268/0.311
Loc 0.350/0.358 0.230/0.251 0.257/0.270 0.210/0.264 0.253/0.265 0.293/0.301 0.199/0.205 0.185/0.189 0.214/0.222 0.203/0.216
Port 0.095/0.111 0.031/0.045 0.021/0.031 0.012/0.023 0.019/0.031 0.008/0.010 0.004/0.004 0.019/0.021 0.010/0.012 0.006/0.006
Fr Rel 0.519/0.548 0.417/0.485 0.509/0.542 0.832/0.890 0.511/0.566 0.530/0.550 0.383/0.440 0.454/0.501 0.827/0.898 0.458/0.506
Gen 0.282/0.305 0.190/0.215 0.219/0.239 0.294/0.297 0.252/0.268 0.281/0.297 0.200/0.222 0.208/0.230 0.308/0.330 0.234/0.281
Loc 0.350/0.362 0.243/0.256 0.249/0.264 0.204/0.217 0.276/0.294 0.303/0.316 0.204/0.214 0.189/0.198 0.214/0.220 0.224/0.208
Port 0.106/0.119 0.020/0.025 0.022/0.023 0.029/0.033 0.023/0.029 0.006/0.018 0.010/0.016 0.010/0.012 0.004/0.006 0.002/0.008
Es Rel 0.528/0.548 0.409/0.458 0.483/0.542 0.489/0.544 0.812/0.908 0.555/0.581 0.391/0.429 0.451/0.516 0.466/0.554 0.822/0.921
Gen 0.297/0.321 0.194/0.217 0.241/0.272 0.231/0.252 0.280/0.315 0.318/0.340 0.184/0.219 0.233/0.251 0.265/0.263 0.330/0.372
Loc 0.346/0.358 0.235/0.250 0.249/0.262 0.209/0.223 0.254/0.268 0.294/0.300 0.211/0.217 0.186/0.188 0.200/0.238 0.211/0.223
Port 0.106/0.123 0.022/0.023 0.035/0.037 0.023/0.025 0.029/0.033 0.008/0.014 0.002/0.002 0.008/0.014 0.010/0.020 0.000/0.002
Table 8: Comparison of reliability (Rel), generalization (Gen), locality (Loc), and portability (Port) scores for multiple language models evaluated using the CounterFact dataset and the ROME editing method. The second column indicates the language in which each model was edited.
Datasets/ Languages Score Mistral TowerInstruct
En De It Fr Es En De It Fr Es
ZSRE En Rel 0.929/0.981 0.344/0.448 0.312/0.344 0.364/0.442 0.390/0.481 0.875/0.928 0.236/0.279 0.240/0.293 0.221/0.260 0.240/0.288
Gen 0.812/0.851 0.260/0.351 0.260/0.325 0.292/0.331 0.331/0.409 0.620/0.683 0.183/0.226 0.168/0.216 0.149/0.207 0.183/0.255
Loc 0.000/0.006 0.013/0.019 0.000/0.019 0.013/0.026 0.013/0.019 0.010/0.019 0.000/0.010 0.000/0.005 0.000/0.014 0.005/0.010
Port 0.097/0.136 0.065/0.071 0.071/0.078 0.058/0.091 0.039/0.058 0.053/0.062 0.019/0.019 0.010/0.024 0.019/0.019 0.019/0.034
De Rel 0.382/0.474 0.513/0.625 0.257/0.336 0.289/0.349 0.270/0.355 0.157/0.216 0.480/0.593 0.221/0.260 0.211/0.240 0.176/0.211
Gen 0.342/0.428 0.349/0.454 0.237/0.309 0.237/0.289 0.217/0.289 0.152/0.196 0.333/0.387 0.162/0.201 0.142/0.172 0.132/0.167
Loc 0.000/0.007 0.013/0.020 0.000/0.013 0.013/0.020 0.013/0.020 0.010/0.020 0.000/0.010 0.000/0.005 0.000/0.015 0.005/0.010
Port 0.079/0.092 0.079/0.099 0.066/0.079 0.072/0.099 0.053/0.086 0.010/0.020 0.025/0.025 0.010/0.015 0.020/0.020 0.010/0.015
It Rel 0.314/0.386 0.288/0.340 0.575/0.654 0.333/0.399 0.281/0.366 0.176/0.224 0.185/0.215 0.537/0.624 0.210/0.268 0.229/0.340
Gen 0.340/0.405 0.242/0.281 0.418/0.484 0.294/0.373 0.222/0.327 0.161/0.200 0.137/0.185 0.346/0.429 0.180/0.239 0.122/0.271
Loc 0.000/0.007 0.013/0.020 0.000/0.020 0.013/0.020 0.013/0.020 0.010/0.015 0.000/0.010 0.000/0.005 0.000/0.015 0.005/0.005
Port 0.059/0.085 0.072/0.078 0.072/0.085 0.078/0.105 0.039/0.072 0.029/0.029 0.029/0.029 0.015/0.020 0.029/0.034 0.020/0.030
Fr Rel 0.424/0.477 0.298/0.344 0.272/0.391 0.517/0.629 0.331/0.444 0.143/0.177 0.153/0.187 0.197/0.256 0.507/0.591 0.138/0.167
Gen 0.371/0.424 0.285/0.325 0.245/0.325 0.404/0.503 0.245/0.351 0.138/0.177 0.133/0.167 0.167/0.192 0.281/0.350 0.113/0.163
Loc 0.000/0.007 0.013/0.020 0.000/0.020 0.013/0.020 0.013/0.020 0.010/0.020 0.000/0.010 0.000/0.005 0.005/0.015 0.005/0.010
Port 0.132/0.159 0.066/0.066 0.073/0.086 0.060/0.093 0.040/0.060 0.015/0.025 0.025/0.025 0.010/0.020 0.034/0.054 0.005/0.020
Es Rel 0.367/0.440 0.260/0.320 0.360/0.433 0.307/0.400 0.487/0.607 0.232/0.232 0.148/0.158 0.241/0.340 0.182/0.236 0.443/0.591
Gen 0.287/0.367 0.227/0.280 0.247/0.313 0.333/0.387 0.353/0.453 0.153/0.177 0.094/0.118 0.202/0.271 0.182/0.241 0.305/0.404
Loc 0.000/0.007 0.013/0.020 0.000/0.020 0.013/0.020 0.007/0.013 0.010/0.010 0.000/0.005 0.000/0.005 0.000/0.010 0.005/0.010
Port 0.060/0.080 0.040/0.060 0.033/0.060 0.047/0.080 0.033/0.067 0.000/0.000 0.010/0.010 0.015/0.030 0.010/0.020 0.020/0.020
Table 9: Comparison of reliability (Rel), generalization (Gen), locality (Loc), and portability (Port) scores for multiple language models evaluated using the ZsRE dataset and the ROME editing method. The second column indicates the language in which each model was edited.
Datasets/ Languages Score Mistral TowerInstruct
En De It Fr Es En De It Fr Es
CounterFact En Rel 0.988/0.988 0.537/0.606 0.438/0.494 0.506/0.588 0.562/0.600 0.954/0.963 0.404/0.459 0.349/0.404 0.450/0.486 0.404/0.477
Gen 0.444/0.456 0.219/0.225 0.212/0.225 0.263/0.269 0.212/0.263 0.431/0.431 0.128/0.174 0.193/0.202 0.193/0.220 0.183/0.220
Loc 0.381/0.388 0.256/0.281 0.275/0.287 0.250/0.263 0.250/0.269 0.275/0.294 0.193/0.220 0.202/0.202 0.193/0.211 0.165/0.165
Port 0.156/0.188 0.025/0.037 0.037/0.037 0.031/0.037 0.025/0.037 0.000/0.000 0.000/0.000 0.009/0.018 0.009/0.009 0.000/0.000
De Rel 0.439/0.484 0.726/0.866 0.376/0.420 0.350/0.369 0.363/0.414 0.355/0.391 0.727/0.827 0.282/0.380 0.309/0.309 0.255/0.300
Gen 0.242/0.280 0.191/0.223 0.185/0.191 0.185/0.217 0.178/0.210 0.227/0.236 0.191/0.218 0.136/0.176 0.182/0.209 0.145/0.164
Loc 0.376/0.389 0.242/0.268 0.280/0.293 0.229/0.242 0.274/0.280 0.264/0.282 0.191/0.218 0.200/0.231 0.209/0.227 0.200/0.200
Port 0.108/0.134 0.045/0.064 0.025/0.025 0.013/0.025 0.032/0.051 0.000/0.000 0.009/0.009 0.009/0.009 0.009/0.009 0.000/0.000
It Rel 0.372/0.404 0.353/0.410 0.801/0.878 0.455/0.526 0.449/0.526 0.407/0.444 0.361/0.380 0.741/0.778 0.389/0.417 0.426/0.454
Gen 0.256/0.263 0.141/0.167 0.237/0.269 0.192/0.231 0.179/0.212 0.315/0.315 0.139/0.176 0.250/0.259 0.204/0.213 0.185/0.213
Loc 0.385/0.397 0.263/0.288 0.269/0.282 0.250/0.263 0.276/0.282 0.269/0.287 0.204/0.231 0.204/0.204 0.194/0.213 0.176/0.176
Port 0.122/0.147 0.013/0.032 0.026/0.026 0.013/0.019 0.019/0.045 0.009/0.009 0.009/0.009 0.019/0.028 0.009/0.009 0.000/0.000
Fr Rel 0.439/0.459 0.395/0.471 0.401/0.433 0.790/0.847 0.446/0.478 0.468/0.477 0.330/0.385 0.330/0.376 0.651/0.716 0.330/0.367
Gen 0.229/0.268 0.153/0.166 0.159/0.172 0.236/0.255 0.153/0.172 0.294/0.312 0.128/0.147 0.183/0.183 0.220/0.239 0.174/0.193
Loc 0.389/0.401 0.268/0.293 0.280/0.293 0.242/0.255 0.274/0.280 0.248/0.266 0.183/0.211 0.183/0.183 0.174/0.193 0.174/0.174
Port 0.089/0.115 0.019/0.032 0.019/0.019 0.025/0.032 0.013/0.025 0.000/0.000 0.009/0.009 0.009/0.018 0.000/0.000 0.000/0.000
Es Rel 0.433/0.465 0.338/0.382 0.401/0.452 0.471/0.522 0.777/0.860 0.435/0.463 0.306/0.324 0.370/0.398 0.380/0.398 0.704/0.796
Gen 0.210/0.229 0.127/0.159 0.121/0.134 0.185/0.217 0.223/0.274 0.241/0.250 0.148/0.157 0.194/0.204 0.213/0.213 0.231/0.269
Loc 0.395/0.408 0.274/0.306 0.268/0.287 0.242/0.255 0.274/0.287 0.259/0.278 0.194/0.222 0.185/0.185 0.176/0.194 0.185/0.185
Port 0.108/0.134 0.025/0.051 0.006/0.006 0.013/0.013 0.025/0.045 0.009/0.009 0.000/0.009 0.009/0.019 0.019/0.019 0.000/0.000
Table 10: Comparison of reliability (Rel), generalization (Gen), locality (Loc), and portability (Port) scores for multiple language models evaluated using the CounterFact dataset and the MEMIT editing method. The second column indicates the language in which each model was edited.
Datasets/ Languages Score Mistral TowerInstruct
En De It Fr Es En De It Fr Es
ZSRE En Rel 0.786/0.812 0.136/0.182 0.227/0.266 0.227/0.279 0.188/0.260 0.528/0.538 0.104/0.142 0.123/0.142 0.123/0.170 0.123/0.142
Gen 0.513/0.545 0.136/0.162 0.175/0.208 0.156/0.208 0.136/0.208 0.321/0.330 0.123/0.142 0.113/0.132 0.075/0.104 0.094/0.113
Loc 0.019/0.026 0.013/0.032 0.013/0.019 0.013/0.019 0.019/0.019 0.019/0.038 0.000/0.019 0.000/0.009 0.000/0.019 0.009/0.019
Port 0.039/0.065 0.019/0.032 0.006/0.006 0.039/0.052 0.039/0.045 0.019/0.028 0.019/0.019 0.019/0.019 0.019/0.019 0.009/0.009
De Rel 0.158/0.204 0.382/0.474 0.138/0.178 0.112/0.132 0.118/0.164 0.029/0.077 0.250/0.298 0.048/0.067 0.038/0.058 0.048/0.048
Gen 0.125/0.171 0.184/0.243 0.138/0.164 0.105/0.118 0.086/0.125 0.058/0.067 0.106/0.115 0.048/0.067 0.038/0.048 0.038/0.058
Loc 0.020/0.026 0.007/0.026 0.013/0.020 0.013/0.020 0.020/0.020 0.019/0.029 0.000/0.010 0.000/0.010 0.000/0.019 0.010/0.019
Port 0.039/0.066 0.020/0.039 0.013/0.013 0.007/0.020 0.020/0.033 0.010/0.019 0.000/0.000 0.000/0.000 0.010/0.010 0.000/0.000
It Rel 0.144/0.176 0.157/0.196 0.425/0.503 0.144/0.183 0.163/0.216 0.019/0.038 0.038/0.067 0.248/0.286 0.067/0.086 0.095/0.124
Gen 0.105/0.150 0.085/0.118 0.255/0.307 0.144/0.183 0.105/0.157 0.029/0.067 0.048/0.076 0.162/0.200 0.038/0.057 0.048/0.067
Loc 0.020/0.026 0.007/0.026 0.013/0.020 0.013/0.020 0.020/0.020 0.019/0.029 0.000/0.019 0.000/0.010 0.000/0.029 0.010/0.019
Port 0.046/0.072 0.007/0.033 0.013/0.033 0.020/0.033 0.020/0.033 0.000/0.010 0.010/0.019 0.019/0.029 0.010/0.010 0.000/0.000
Fr Rel 0.139/0.172 0.099/0.152 0.166/0.238 0.397/0.497 0.119/0.166 0.048/0.077 0.048/0.067 0.038/0.077 0.269/0.346 0.019/0.058
Gen 0.152/0.212 0.079/0.139 0.139/0.185 0.185/0.272 0.093/0.139 0.019/0.038 0.029/0.048 0.048/0.077 0.144/0.173 0.010/0.019
Loc 0.020/0.026 0.013/0.033 0.013/0.020 0.013/0.020 0.020/0.020 0.019/0.029 0.000/0.019 0.000/0.010 0.000/0.019 0.010/0.010
Port 0.060/0.079 0.020/0.033 0.020/0.020 0.040/0.060 0.040/0.053 0.019/0.019 0.010/0.010 0.000/0.010 0.029/0.029 0.000/0.000
Es Rel 0.107/0.153 0.073/0.106 0.166/0.213 0.147/0.186 0.373/0.493 0.058/0.087 0.038/0.058 0.087/0.115 0.058/0.106 0.240/0.337
Gen 0.087/0.256 0.087/0.106 0.140/0.173 0.093/0.146 0.220/0.286 0.048/0.087 0.058/0.087 0.087/0.115 0.058/0.087 0.163/0.202
Loc 0.020/0.026 0.007/0.026 0.013/0.020 0.013/0.020 0.020/0.020 0.019/0.029 0.000/0.019 0.000/0.010 0.000/0.019 0.010/0.019
Port 0.033/0.060 0.007/0.013 0.027/0.033 0.033/0.046 0.027/0.040 0.010/0.010 0.000/0.000 0.010/0.010 0.019/0.019 0.010/0.019
Table 11: Comparison of reliability (Rel), generalization (Gen), locality (Loc), and portability (Port) scores for multiple language models evaluated using the ZsRE dataset and the MEMIT editing method. The second column indicates the language in which each model was edited.