[Paper Review] Helping results assessment by adding explainable elements to the deep relevance matching model

Ioannis Chios, Suzan Verberne|arXiv (Cornell University)|Jun 9, 2021

Topic Modeling22 references24 citations

TL;DR

The paper adds explainable elements to a DRMM-based search interface—query-term weights visualized as a doughnut chart and a passage-level snippet with highlighted passages and thumbnails—and shows via user study that this interface is more explainable and assessable, though not clearly better at selecting top-5 relevant documents.

ABSTRACT

In this paper we address the explainability of web search engines. We propose two explainable elements on the search engine result page: a visualization of query term weights and a visualization of passage relevance. The idea is that search engines that indicate to the user why results are retrieved are valued higher by users and gain user trust. We deduce the query term weights from the term gating network in the Deep Relevance Matching Model (DRMM) and visualize them as a doughnut chart. In addition, we train a passage-level ranker with DRMM that selects the most relevant passage from each document and shows it as snippet on the result page. Next to the snippet we show a document thumbnail with this passage highlighted. We evaluate the proposed interface in an online user study, asking users to judge the explainability and assessability of the interface. We found that users judge our proposed interface significantly more explainable and easier to assess than a regular search engine result page. However, they are not significantly better in selecting the relevant documents from the top-5. This indicates that the explainability of the search engine result page leads to a better user experience. Thus, we conclude that the proposed explainable elements are promising as visualization for search engine users.

Motivation & Objective

• Motivate explainability in non-personalized web search and increase user trust by making relevance signals visible.
• Leverage DRMM to extract per-term importance and passage-level relevance for visual explanations.
• Evaluate the impact of explainable visualizations on user judgments of explainability, assessability, and relevance selection.
• Investigate whether explainable elements affect users' ability to identify relevant documents from snippets.

Proposed method

• Use Anserini BM25 to retrieve top K documents and then re-rank with DRMM (MatchZoo implementation).
• Adapt DRMM to produce term-level importance via the term gating network, interpreted as query term weights for visualization.
• Split documents into non-overlapping 100-token passages and train a passage-level DRMM to rank passages within documents (assigning passage scores and using the maxP passage as the document snippet).
• Visualize query-term importance as a doughnut chart on the result page and display a document thumbnail with the highest-scoring passage highlighted as the snippet.
• Train using hinge loss with pairwise ranking and Adadelta optimization; use LogCount-based histograms for local interactions in DRMM.
• Evaluate with Robust04 data, 5-fold cross-validation for DRMM, and a user study comparing regular vs. explainable interfaces.

Experimental results

Research questions

• RQ1 What is the ranking effectiveness of DRMM when selecting the most relevant passage of each document?
• RQ2 How do users judge the explainability and assessability of the explainable search results page compared to a regular page?
• RQ3 How well can users select relevant documents based on only the snippets in the explainable interface versus the regular interface?

Key findings

• The explainable interface is judged significantly more explainable than the regular interface (mean explainability 4.2 vs 3.4).
• The explainable interface is judged significantly more assessable than the regular interface (mean assessability 4.4 vs 3.6).
• Per-interface analyses show explainability differences are robust across participants and queries (significant interface effect, p<0.001 in ANOVA).
• Users’ precision and recall on relevance judgments from the result page alone did not show a significant improvement with the explainable interface (p=0.90 for precision, p=0.38 for recall).
• In retrieval performance, the passage-level ranking model is less effective than BM25 and document-level DRMM on P@20 and nDCG@20, but slightly better on MAP, indicating acceptable quality for evaluating explainable features.