[Paper Review] Recommender Systems for Configuration Knowledge Engineering

Alexander Felfernig, Stefan Reiterer+4 more|arXiv (Cornell University)|Feb 16, 2021

Product Development and CustomizationBusiness, Management and Accounting17 references19 citations

TL;DR

This paper proposes integrating recommender systems into configuration knowledge engineering to alleviate the knowledge engineering bottleneck by supporting knowledge base development and maintenance. It introduces collaborative filtering, content-based clustering, and knowledge-based recommendations for constraints, variables, and refactoring operations, with empirical results showing that implication-based representations (e.g., X→Y) significantly improve understanding and reduce errors compared to alternative logical forms.

ABSTRACT

The knowledge engineering bottleneck is still a major challenge in configurator projects. In this paper we show how recommender systems can support knowledge base development and maintenance processes. We discuss a couple of scenarios for the application of recommender systems in knowledge engineering and report the results of empirical studies which show the importance of user-centered configuration knowledge organization.

Motivation & Objective

• Address the persistent knowledge engineering bottleneck in configurator projects by improving efficiency in knowledge base development and maintenance.
• Support knowledge engineers and domain experts in navigating, understanding, and debugging complex configuration knowledge bases.
• Investigate how recommender systems can proactively suggest relevant configuration elements (e.g., constraints, variables, refactoring operations) based on user behavior and knowledge structure.
• Evaluate the impact of different logical representations (e.g., implications vs. disjunctions) on user comprehension and error rates in configuration tasks.
• Advance the field of empirical knowledge engineering by grounding recommendations in cognitive processes and usability studies.

Proposed method

• Apply collaborative filtering to recommend constraints based on navigation patterns of similar knowledge engineers, using user access sequences to identify nearest neighbors and predict next likely actions.
• Implement content-based clustering of constraints using similarity metrics derived from feature descriptors (e.g., keywords, logical structure) to group semantically or structurally similar constraints.
• Develop knowledge-based recommenders that suggest refactoring operations by analyzing constraint patterns and applying domain-specific rules or transformation heuristics.
• Design controlled empirical studies with computer science students to evaluate the usability and cognitive load of different constraint representations in constraint satisfaction problems.
• Use a within-subjects design to compare error rates and solution identification performance across five different logical representations of requires and incompatibility constraints.
• Integrate adaptive user interface techniques that tailor recommendations to the user’s current task, knowledge level, and interaction history in the knowledge engineering environment.

Experimental results

Research questions

• RQ1 How effective are collaborative filtering recommender systems in suggesting relevant configuration constraints based on user navigation behavior?
• RQ2 To what extent does content-based clustering of constraints improve the organization and navigability of large configuration knowledge bases?
• RQ3 Which logical representation of requires and incompatibility constraints (e.g., X→Y, ¬X∨Y, ¬(X∧¬Y)) leads to the highest comprehension and lowest error rates in configuration tasks?
• RQ4 How do different constraint representations affect the time and accuracy of solution identification in constraint satisfaction problems?
• RQ5 Can knowledge-based recommendations for refactoring operations reduce maintenance effort and improve knowledge base quality in configuration systems?

Key findings

• The implication-based representation X→Y achieved the lowest error rate (21.43%) in identifying solutions for requires constraints, outperforming other forms such as ¬X∨Y (50.0%) and ¬Y→¬X (96.43%).
• The reverse implication Y→¬X showed moderate performance (50.0% errors), but the standard implication X→Y was judged superior due to higher understandability and lower cognitive load.
• Content-based clustering of constraints was found to be effective in grouping semantically similar constraints, improving navigability and reducing search time in large knowledge bases.
• Collaborative filtering recommendations based on navigation sequences successfully predicted the next likely constraint to examine, with high consistency among nearest neighbors.
• The use of multiple constraint representations in the same knowledge base led to significant confusion, with error rates increasing by up to 73.08% when using complex logical forms like ¬(X∧¬Y).
• Empirical results confirmed that basic implication representations (→) are more intuitive and less error-prone than disjunctive or negated forms, supporting their adoption in knowledge engineering tools.