[Paper Review] Delphic Costs and Benefits in Web Search: A utilitarian and historical analysis

Andrei Broder, Preston McAfee|arXiv (Cornell University)|Aug 15, 2023

Digital Marketing and Social MediaSocial Sciences3 citations

TL;DR

This paper introduces the concept of 'Delphic costs and benefits'—non-monetary user experience factors in web search such as cognitive load, time, interactivity, and risks like misinformation—to frame search engine utility holistically. It argues that search engine satisfaction stems primarily from minimizing these costs and maximizing benefits, and traces how advancements from classic IR to LLM-powered chatbots have been driven by this goal.

ABSTRACT

We present a new framework to conceptualize and operationalize the total user experience of search, by studying the entirety of a search journey from an utilitarian point of view. Web search engines are widely perceived as "free". But search requires time and effort: in reality there are many intermingled non-monetary costs (e.g. time costs, cognitive costs, interactivity costs) and the benefits may be marred by various impairments, such as misunderstanding and misinformation. This characterization of costs and benefits appears to be inherent to the human search for information within the pursuit of some larger task: most of the costs and impairments can be identified in interactions with any web search engine, interactions with public libraries, and even in interactions with ancient oracles. To emphasize this innate connection, we call these costs and benefits Delphic, in contrast to explicitly financial costs and benefits. Our main thesis is that the users' satisfaction with a search engine mostly depends on their experience of Delphic cost and benefits, in other words on their utility. The consumer utility is correlated with classic measures of search engine quality, such as ranking, precision, recall, etc., but is not completely determined by them. To argue our thesis, we catalog the Delphic costs and benefits and show how the development of search engines over the last quarter century, from classic Information Retrieval roots to the integration of Large Language Models, was driven to a great extent by the quest of decreasing Delphic costs and increasing Delphic benefits. We hope that the Delphic costs framework will engender new ideas and new research for evaluating and improving the web experience for everyone.

Motivation & Objective

• To reframe web search evaluation beyond traditional metrics like precision and recall by incorporating the full spectrum of user experience costs and benefits.
• To identify and categorize non-monetary costs—cognitive, time, interactivity, privacy, and impairments like misinformation—that affect search utility.
• To draw historical parallels between ancient oracular consultation and modern web search, emphasizing shared user experience challenges.
• To analyze how the evolution of search engines, especially the rise of LLM-powered chatbots, has been driven by the goal of reducing Delphic costs and increasing Delphic benefits.
• To advocate for a holistic, utilitarian evaluation framework applicable not only to search but also to content feeds and recommender systems.

Proposed method

• Proposes a new conceptual framework—'Delphic costs and benefits'—to model the total user experience in web search, drawing analogies to ancient oracular consultation.
• Categorizes Delphic costs into access, cognitive, interactivity, time, privacy trade-offs, and impairments (miscommunication, misrepresentation, misinformation, disinformation, misinterpretation, safety flaws).
• Analyzes the impact of search engine evolution—from classic IR to modern LLM chatbots—through the lens of Delphic cost reduction and benefit enhancement.
• Evaluates chatbots based on their ability to reduce query crafting and result processing effort, improve task completion, and support natural language and voice interaction.
• Assesses trade-offs such as increased latency, access costs, and hallucination risks that may offset gains in utility.
• Proposes that future evaluation of search systems should prioritize user utility over isolated ranking quality, integrating context, intent, and end-task completion.

Experimental results

Research questions

• RQ1 How do non-monetary costs and benefits—such as cognitive load, time, and misinformation—shape user satisfaction in web search?
• RQ2 To what extent do historical parallels between ancient oracles and modern search engines illuminate persistent challenges in information retrieval?
• RQ3 How have the development of search engines over the past 25 years been driven by the goal of minimizing Delphic costs and maximizing Delphic benefits?
• RQ4 In what ways do LLM-powered chatbots reduce or increase Delphic costs and benefits compared to traditional search engines?
• RQ5 Can a utilitarian, holistic evaluation framework based on Delphic costs and benefits replace or augment traditional metrics like precision and recall in search evaluation?

Key findings

• Delphic costs—cognitive, time, interactivity, privacy, and impairments such as misinformation—are central to user satisfaction and are inherent to information-seeking across eras, from ancient oracles to modern search.
• The rise of LLM-powered chatbots has significantly reduced query crafting and result processing costs by enabling natural language interaction and synthesized responses, improving task completion efficiency.
• Chatbots reduce Delphic costs in query formulation and result consumption, especially for complex or technical tasks, such as generating code or writing narratives.
• Despite benefits, chatbots increase latency, access costs, and the risk of hallucinations and misinformation, which can undermine trust and require verification, potentially nullifying time savings.
• The polished output of chatbots may create a false sense of reliability, as seen in cases where experts accepted citations to non-existent legal cases.
• A holistic evaluation of search systems must move beyond ranking quality to include user context, task completion, and the full spectrum of Delphic costs and benefits to reflect true utility.