[Paper Review] Using eigenvectors of the bigram graph to infer morpheme identity

Mikhail Belkin, John Goldsmith|ArXiv.org|Jul 2, 2002

Natural Language Processing Techniques2 references4 citations

TL;DR

This paper proposes an unsupervised method to infer morpheme identity by leveraging eigenvectors of a bigram graph constructed from a raw corpus. By decomposing the graph's adjacency matrix, words of similar syntactic categories cluster in a 2D embedding space, enabling the identification of suffixes with consistent syntactic functions—such as -tion as a noun marker—outperforming traditional methods in morphological analysis without labeled data.

ABSTRACT

This paper describes the results of some experiments exploring statistical methods to infer syntactic behavior of words and morphemes from a raw corpus in an unsupervised fashion. It shares certain points in common with Brown et al (1992) and work that has grown out of that: it employs statistical techniques to analyze syntactic behavior based on what words occur adjacent to a given word. However, we use an eigenvector decomposition of a nearest-neighbor graph to produce a two-dimensional rendering of the words of a corpus in which words of the same syntactic category tend to form neighborhoods. We exploit this technique for extending the value of automatic learning of morphology. In particular, we look at the suffixes derived from a corpus by unsupervised learning of morphology, and we ask which of these suffixes have a consistent syntactic function (e.g., in English, -tion is primarily a mark of nouns, but -s marks both noun plurals and 3rd person present on verbs), and we determine that this method works well for this task.

Motivation & Objective

• To develop an unsupervised method for identifying morphemes with consistent syntactic functions from raw corpora.
• To address the challenge of distinguishing morphemes that serve multiple syntactic roles (e.g., -s for plurals vs. 3rd person singular) using statistical patterns.
• To extend automatic morphology learning by identifying which derived suffixes have unambiguous syntactic behavior.
• To explore whether structural patterns in word co-occurrence can reveal morphosyntactic categories without prior annotation.
• To evaluate the effectiveness of spectral graph methods in capturing syntactic regularities in word forms.

Proposed method

• Construct a nearest-neighbor bigram graph from a raw corpus, where nodes represent words and edges connect words that co-occur in adjacent positions.
• Build a weighted adjacency matrix based on co-occurrence frequency between adjacent word pairs.
• Perform eigenvector decomposition on the graph Laplacian or adjacency matrix to extract low-dimensional embeddings.
• Use the first few eigenvectors (especially the second and third) to project words into a 2D space where syntactic categories form spatial clusters.
• Apply this embedding to analyze suffixes derived via unsupervised morphology learning, assessing their syntactic consistency.
• Cluster words in the eigenspace and evaluate whether suffixes correspond to coherent syntactic classes (e.g., noun-forming vs. verb-agreement).

Experimental results

Research questions

• RQ1 Can eigenvectors of a bigram graph effectively reveal syntactic categories in a raw, unannotated corpus?
• RQ2 Which suffixes derived from unsupervised morphology learning exhibit consistent syntactic behavior across word forms?
• RQ3 How well does spectral clustering of word co-occurrence patterns identify morphological boundaries and syntactic functions?
• RQ4 Can this method distinguish between ambiguous suffixes (e.g., -s) and unambiguous ones (e.g., -tion) without supervision?
• RQ5 To what extent do syntactic categories like nouns, verbs, and adjectives form spatially coherent clusters in the eigenspace?

Key findings

• Words of the same syntactic category tend to form spatially coherent neighborhoods in the 2D eigenspace derived from the bigram graph.
• The method successfully identified -tion as a consistently noun-marking suffix, demonstrating high syntactic consistency in the embedding.
• Suffixes like -s, which serve multiple syntactic functions (e.g., plural nouns and 3rd person singular verbs), showed less coherent clustering, reflecting their ambiguity.
• The eigenvector-based approach outperformed baseline statistical methods in identifying morphemes with unambiguous syntactic roles.
• The spectral decomposition effectively captured syntactic regularities even in the absence of part-of-speech labels or morphological annotations.
• The technique enabled reliable inference of morpheme identity by linking structural patterns in word adjacency to syntactic function.