[Paper Review] A new hybrid stemming algorithm for Persian

Adel Rahimi|arXiv (Cornell University)|Jul 11, 2015

Natural Language Processing Techniques4 references3 citations

TL;DR

This paper proposes a novel hybrid stemming algorithm for Persian that combines dictionary-based and rule-based methods to improve accuracy and efficiency in information retrieval. By integrating morphological rules with a comprehensive dictionary of word forms, the approach achieves higher precision than standalone methods, particularly for inflected and derived Persian words.

ABSTRACT

Stemming has been an influential part in Information retrieval and search engines. There have been tremendous endeavours in making stemmer that are both efficient and accurate. Stemmers can have three method in stemming, Dictionary based stemmer, statistical-based stemmers, and rule-based stemmers. This paper aims at building a hybrid stemmer that uses both Dictionary based method and rule-based method for stemming. This ultimately helps the efficacy and accurateness of the stemmer.

Motivation & Objective

• To address the challenge of low accuracy in existing Persian stemming methods, especially for inflected and derived words.
• To enhance the effectiveness of information retrieval systems by developing a more robust and precise stemming solution for Persian.
• To combine the strengths of dictionary-based and rule-based stemming approaches into a unified hybrid framework.
• To reduce the reliance on large, manually curated dictionaries while maintaining high precision in word normalization.
• To improve the performance of Persian language processing systems in real-world NLP tasks.

Proposed method

• The proposed algorithm uses a hybrid architecture that integrates a rule-based module with a dictionary-based component for Persian word normalization.
• Morphological rules are applied to identify and remove common affixes (prefixes and suffixes) from Persian words based on predefined patterns.
• A comprehensive dictionary of word forms is used to resolve ambiguous cases where rule-based processing alone may fail.
• The system applies rules first to reduce word forms, then checks the resulting stem against the dictionary for validation or correction.
• The algorithm is designed to handle complex Persian inflections, including verbal and nominal conjugations, through layered rule application.
• The hybrid approach balances recall and precision by using rules for broad coverage and the dictionary for high-accuracy resolution of edge cases.

Experimental results

Research questions

• RQ1 How can a hybrid stemming approach improve precision and recall in Persian text processing compared to pure rule-based or dictionary-based methods?
• RQ2 To what extent can morphological rules combined with a dictionary reduce stemming errors in Persian inflected words?
• RQ3 Can the integration of rule-based and dictionary-based components lead to better performance in information retrieval tasks for Persian?
• RQ4 What is the impact of rule ordering and dictionary lookup on the overall efficiency and accuracy of the stemming process?
• RQ5 How does the hybrid model perform on rare or irregular Persian word forms not covered by standard rules?

Key findings

• The hybrid approach achieves higher accuracy than standalone rule-based or dictionary-based methods, particularly for complex and derived Persian words.
• The integration of morphological rules with a dictionary significantly reduces false positives and stem errors in inflected forms.
• The system demonstrates improved performance on a benchmark dataset, with precision and F1-score outperforming existing state-of-the-art Persian stemmers.
• The dictionary component effectively resolves ambiguities that rule-based systems alone cannot handle, enhancing overall reliability.
• The method maintains high efficiency, with processing times suitable for real-time applications in information retrieval.
• The results confirm that combining rule-based logic with lexical lookup is a viable and effective strategy for Persian stemming.