[Paper Review] Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons

Yuheng Chen, Pengfei Cao+3 more|arXiv (Cornell University)|Aug 25, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes Architecture-adapted Multilingual Integrated Gradients (AMIG), a universal method for precisely localizing knowledge neurons across diverse PLM architectures and languages. It discovers two key phenomena: language-independent knowledge neurons that store cross-lingual facts and degenerate knowledge neurons that enable robust fact-checking through functional redundancy, significantly improving factual consistency in multilingual models.

ABSTRACT

Pre-trained language models (PLMs) contain vast amounts of factual knowledge, but how the knowledge is stored in the parameters remains unclear. This paper delves into the complex task of understanding how factual knowledge is stored in multilingual PLMs, and introduces the Architecture-adapted Multilingual Integrated Gradients method, which successfully localizes knowledge neurons more precisely compared to current methods, and is more universal across various architectures and languages. Moreover, we conduct an in-depth exploration of knowledge neurons, leading to the following two important discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which store factual knowledge in a form that transcends language. We design cross-lingual knowledge editing experiments, demonstrating that the PLMs can accomplish this task based on language-independent neurons; (2) The discovery of Degenerate Knowledge Neurons, a novel type of neuron showing that different knowledge neurons can store the same fact. Its property of functional overlap endows the PLMs with a robust mastery of factual knowledge. We design fact-checking experiments, proving that the degenerate knowledge neurons can help the PLMs to detect wrong facts. Experiments corroborate these findings, shedding light on the mechanisms of factual knowledge storage in multilingual PLMs, and contribute valuable insights to the field. The code is available at https://github.com/heng840/AMIG.

Motivation & Objective

• To address the lack of a universal method for localizing knowledge neurons across different PLM architectures (e.g., BERT vs. GPT).
• To explore factual knowledge storage mechanisms in multilingual pre-trained language models beyond monolingual English-focused studies.
• To identify and validate novel types of knowledge neurons that transcend language boundaries and exhibit functional redundancy.
• To improve cross-lingual knowledge editing and fact-checking performance using localized neurons.
• To provide deeper insight into the structural and functional mechanisms of knowledge storage in multilingual PLMs.

Proposed method

• Propose AMIG, a gradient-based method that adapts baseline vectors in integrated gradients to be universally compatible across auto-encoding and auto-regressive PLM architectures.
• Introduce an architecture adaptation technique that ensures consistent attribution scores regardless of model architecture.
• Implement a multilingual threshold adjustment technique to optimize integrated gradient calculations across different languages.
• Use intersection-based neuron clustering across languages to identify language-independent knowledge neurons.
• Apply causal tracing and fact-checking benchmarks to validate the functional roles of discovered neurons.
• Leverage integrated gradients with axiomatic attribution principles to ensure sensitivity and implementation invariance in neuron attribution.

Experimental results

Research questions

• RQ1 Can knowledge neurons be localized with high precision across diverse PLM architectures, including both auto-encoding and auto-regressive models?
• RQ2 Do factual knowledge representations in multilingual PLMs exist in a form that transcends specific languages, enabling cross-lingual editing?
• RQ3 Is there a functional redundancy in knowledge neurons such that multiple neurons store the same fact, enhancing model robustness?
• RQ4 Can degenerate knowledge neurons be leveraged to improve fact-checking performance in multilingual models?
• RQ5 Does the presence of degenerate knowledge neurons contribute to improved model reliability in detecting incorrect facts?

Key findings

• AMIG localizes knowledge neurons with higher precision than prior state-of-the-art methods across multilingual datasets and diverse PLM architectures.
• Language-independent knowledge neurons exist and can be edited to simultaneously update the same factual knowledge across all supported languages, enabling efficient cross-lingual knowledge editing.
• Degenerate knowledge neurons—functionally overlapping neurons storing the same fact—were discovered, enhancing model robustness against factual errors.
• In fact-checking tasks, using degenerate knowledge neurons improved F1 scores by up to 167,150% over baseline methods in Chinese m-GPT, demonstrating strong detection capability.
• The existence of degenerate knowledge neurons is independent of multilingualism, as they were also observed in monolingual models like BERT and GPT-2.
• Auto-regressive models showed higher Recall in fact-checking due to their coherence-focused design, but degenerate neurons significantly improved F1 across all models, including BERT and GPT-2.