[論文レビュー] Multiplicative Orthogonal Sequential Editing for Language Models

Hao-Xiang Xu, Jun-Yu Ma+4 more|arXiv (Cornell University)|Jan 11, 2026

Topic Modeling被引用数 0

ひとこと要約

tldr: MOSE introduces an orthogonal, multiplicative editing paradigm that preserves numerical stability while updating knowledge in LLMs, improving sequential editing performance and maintaining general abilities.

ABSTRACT

Knowledge editing aims to efficiently modify the internal knowledge of large language models (LLMs) without compromising their other capabilities. The prevailing editing paradigm, which appends an update matrix to the original parameter matrix, has been shown by some studies to damage key numerical stability indicators (such as condition number and norm), thereby reducing editing performance and general abilities, especially in sequential editing scenario. Although subsequent methods have made some improvements, they remain within the additive framework and have not fundamentally addressed this limitation. To solve this problem, we analyze it from both statistical and mathematical perspectives and conclude that multiplying the original matrix by an orthogonal matrix does not change the numerical stability of the matrix. Inspired by this, different from the previous additive editing paradigm, a multiplicative editing paradigm termed Multiplicative Orthogonal Sequential Editing (MOSE) is proposed. Specifically, we first derive the matrix update in the multiplicative form, the new knowledge is then incorporated into an orthogonal matrix, which is multiplied by the original parameter matrix. In this way, the numerical stability of the edited matrix is unchanged, thereby maintaining editing performance and general abilities. We compared MOSE with several current knowledge editing methods, systematically evaluating their impact on both editing performance and the general abilities across three different LLMs. Experimental results show that MOSE effectively limits deviations in the edited parameter matrix and maintains its numerical stability. Compared to current methods, MOSE achieves a 12.08% improvement in sequential editing performance, while retaining 95.73% of general abilities across downstream tasks. The code is available at https://github.com/famoustourist/MOSE.

研究の動機と目的

• Objective1: Address the instability of additive knowledge editing in sequential updates.
• Objective2: Propose a multiplicative editing framework using orthogonal transformations to preserve norm and condition number.
• Objective3: Show that MOSE maintains editing performance while retaining general abilities across downstream tasks.
• Objective4: Evaluate MOSE against state-of-the-art methods on multiple LLMs and editing datasets.

提案手法

• Method1: Replace additive updates with left-multiplication by an orthogonal update matrix on the original parameter matrix.
• Method2: Formulate the update as a constrained least-squares (Orthogonal Procrustes) problem to find the optimal orthogonal transform.
• Method3: Minimize a combination of preserving original knowledge and fitting new knowledge via a regularized objective.
• Method4: Select editing layers using a layer activation-based criterion and extend edits to neighboring layers for improved performance.
• Method5: Provide analytic proofs that left-multiplication by an orthogonal matrix preserves Frobenius norm and matrix condition number.

実験結果

リサーチクエスチョン

• RQ1 Research Question1: Can orthogonal, multiplicative updates preserve numerical stability during sequential editing?
• RQ2 Research Question2: Does MOSE maintain editing performance and general abilities better than additive methods under sequential and batch editing?
• RQ3 Research Question3: How should editing layers be selected to maximize MOSE effectiveness across knowledge updates?

主な発見

Method	Model	CounterFact Reliability	CounterFact Generalization	CounterFact Locality	ConceptEdit-Inter Reliability	ConceptEdit-Inter Generalization	ConceptEdit-Inter Locality
ROME	LLama3-8B	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
MEMIT	LLama3-8B	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
RECT	LLama3-8B	0.5266	0.3075	0.2382	0.3234	0.1993	0.1397
EMMET	LLama3-8B	0.6287	0.4695	0.3114	0.3866	0.2178	0.1563
PRUNE	LLama3-8B	0.7738	0.6899	0.5190	0.5682	0.4097	0.3083
AlphaEdit	LLama3-8B	0.8222	0.7835	0.7091	0.6981	0.5928	0.4977
MOSE	LLama3-8B	0.9887	0.9863	0.8972	0.7859	0.7275	0.6856
ROME	Qwen2.5-7B	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
MEMIT	Qwen2.5-7B	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
RECT	Qwen2.5-7B	0.6203	0.4745	0.3582	0.3737	0.2306	0.1738
EMMET	Qwen2.5-7B	0.6702	0.5589	0.4771	0.4593	0.2641	0.1903
PRUNE	Qwen2.5-7B	0.8115	0.7860	0.6823	0.6708	0.5009	0.4120
AlphaEdit	Qwen2.5-7B	0.9519	0.9241	0.8418	0.7346	0.6453	0.6116
MOSE	Qwen2.5-7B	0.9981	0.9902	0.9098	0.8012	0.7547	0.7069

• Key Finding1: MOSE preserves the edited parameter matrix's numerical stability (norm and condition number) during sequential edits.
• Key Finding2: MOSE achieves a 12.08% improvement in sequential editing performance over baseline additive methods.
• Key Finding3: MOSE retains 95.73% of the model’s general abilities on downstream tasks across experiments.
• Key Finding4: In batch-sequential editing, MOSE consistently outperforms baselines and scales better.
• Key Finding5: Layer-aware MOSE (editing selected layer and neighbors) yields the best results in sequential and batch settings.