[Paper Review] A Morphological Analyzer for Japanese Nouns, Verbs and Adjectives

Yanchuan Sim|arXiv (Cornell University)|Oct 1, 2014

Natural Language Processing Techniques4 references3 citations

TL;DR

This paper presents an open-source morphological analyzer for Japanese nouns, verbs, and adjectives that extends MeCab with fine-grained morphological features such as politeness, tense, mood, and voice using finite state transducers (FSTs) in the FOMA toolkit. It achieves 89.2% recall and 89.2% precision for adjectives, and 89.2% recall with 77.6% precision for verbs, significantly improving on MeCab’s coarse-grained analysis by capturing honorifics, collective suffixes, and complex verbal conjugations.

ABSTRACT

We present an open source morphological analyzer for Japanese nouns, verbs and adjectives. The system builds upon the morphological analyzing capabilities of MeCab to incorporate finer details of classification such as politeness, tense, mood and voice attributes. We implemented our analyzer in the form of a finite state transducer using the open source finite state compiler FOMA toolkit. The source code and tool is available at https://bitbucket.org/skylander/yc-nlplab/.

Motivation & Objective

• To develop a morphological analyzer that captures fine-grained grammatical features—such as politeness, tense, mood, and voice—beyond MeCab’s standard 69 POS tags.
• To address the lack of detailed morphological analysis in existing tools, particularly for honorifics, collective suffixes, and verbal conjugations in Japanese.
• To improve morphological analysis precision and recall for verbs and adjectives by extending MeCab’s output with a finite state transducer (FST) framework.
• To evaluate the system on real-world corpora (Tanaka and Japanese-Multilingual Dictionary) and identify key failure modes and coverage gaps.
• To propose post-processing heuristics that reduce over-generation while preserving recall, enhancing practical utility.

Proposed method

• The analyzer uses the FOMA finite state compiler to implement a finite state transducer (FST) that processes Japanese word forms and generates multiple morphological analyses.
• It extends MeCab’s output by adding detailed morphological features such as politeness (o-/go-), voice (passive), aspect (perfective), and mood (potential, negative).
• The system leverages the Japanese-Multilingual Dictionary and Tanaka corpus to build a lexicon of verbs and adjectives with full conjugation patterns.
• A post-processing recognizer is applied to filter invalid morpheme orderings and align analyses with MeCab’s dictionary forms, reducing over-generation.
• The FST is trained and evaluated on 2,000 verb and 1,000 adjective tokens from the Tanaka corpus, with precision and recall computed per token.
• The system handles both i-adjectives (ending in -i) and na-adjectives (requiring the copula -da), treating the copula as an irregular verb for consistent analysis.

Experimental results

Research questions

• RQ1 Can a finite state transducer effectively model fine-grained morphological features—such as politeness, tense, and voice—in Japanese verbs and adjectives beyond MeCab’s standard POS tags?
• RQ2 How does the system perform in capturing honorific prefixes (o-/go-) and suffixes (e.g., -san, -kun, -sama) in noun and name forms?
• RQ3 What is the impact of lexical coverage and corpus type on precision and recall, especially given the agglutinative nature of Japanese and high productivity of verb conjugations?
• RQ4 To what extent can post-processing heuristics reduce over-generation without sacrificing recall in morphological analysis?
• RQ5 How do the analysis results differ between i-adjectives and na-adjectives in terms of conjugation complexity and performance metrics?

Key findings

• The analyzer achieved 89.2% recall and 89.2% precision for adjectives, indicating strong performance on a class with fewer conjugations and more stable forms.
• For verbs, the system achieved 89.2% recall but only 77.6% precision, reflecting challenges from high morphological productivity and colloquial forms in the Tanaka corpus.
• The system successfully captured complex verbal forms such as ‘信じられている’ (had been believed), generating both main and less common analyses via FST rules.
• The post-processing step significantly reduced over-generation by filtering invalid morpheme sequences and aligning with MeCab’s lemmas, improving precision without harming recall.
• Coverage gaps were most pronounced for verbs due to their high productivity and limited dictionary coverage, especially in conversational and informal language.
• 77.4% of the adjectives in the evaluation set were in their base (dictionary) form, which contributed to higher precision compared to the more conjugated verb forms.