[Paper Review] Decentralized Search on Decentralized Web

Ziliang Lai, Chris Liu+3 more|arXiv (Cornell University)|Aug 18, 2018

Caching and Content Delivery6 references4 citations

TL;DR

This paper proposes QueenBee, the first decentralized search engine for the Decentralized Web (DWeb), which eliminates centralized crawling by incentivizing content providers and peer nodes via a blockchain-based smart contract system. It uses cryptocurrency rewards ('honey') to maintain fresh indexes and secure rankings through peer participation, enabling a trustless, censorship-resistant search ecosystem.

ABSTRACT

Decentralized Web, or DWeb, is envisioned as a promising future of the Web. Being decentralized, there are no dedicated web servers in DWeb; Devices that retrieve web contents also serve their cached data to peer devices with straight privacy-preserving mechanisms. The fact that contents in DWeb are distributed, replicated, and decentralized lead to a number of key advantages over the conventional web. These include better resiliency against network partitioning and distributed-denial-of-service attacks (DDoS), and better browsing experiences in terms of shorter latency and higher throughput. Moreover, DWeb provides tamper-proof contents because each content piece is uniquely identified by a cryptographic hash. DWeb also clicks well with future Internet architectures, such as Named Data Networking (NDN).Search engines have been an inseparable element of the Web. Contemporary ("Web 2.0") search engines, however, provide centralized services. They are thus subject to DDoS attacks, insider threat, and ethical issues like search bias and censorship. As the web moves from being centralized to being decentralized, search engines ought to follow. We propose QueenBee, a decentralized search engine for DWeb. QueenBee is so named because worker bees and honeycomb are a common metaphor for distributed architectures, with the queen being the one that holds the colony together. QueenBee aims to revolutionize the search engine business model by offering incentives to both content providers and peers that participate in QueenBee's page indexing and ranking operations.

Motivation & Objective

• To address the limitations of centralized search engines, including censorship, search bias, and vulnerability to DDoS and insider attacks.
• To enable a decentralized, privacy-preserving search infrastructure that aligns with the principles of the Decentralized Web (DWeb).
• To design a self-governing, incentive-driven ecosystem where content providers and peer nodes are rewarded for contributing to indexing and ranking.
• To mitigate risks inherent in decentralized systems, such as collusion and scraping attacks, through cryptographic and economic mechanisms.
• To pioneer a new model for search that integrates with emerging protocols like IPFS and Named Data Networking (NDN).

Proposed method

• QueenBee leverages smart contracts deployed on a blockchain (e.g., Ethereum) to autonomously govern indexing, ranking, and reward distribution.
• Content providers publish or update content directly via the smart contract to earn 'honey'—a native cryptocurrency token—as incentive.
• Peer nodes (worker bees) are rewarded for maintaining the decentralized index and computing page ranks using data stored in IPFS.
• Search queries are submitted through a DWeb-hosted HTML+JavaScript frontend that composes results by intersecting inverted lists and ranking them.
• Advertisers directly interact with the smart contract, and ad revenue is shared among content creators and worker bees based on performance metrics.
• The system ensures data integrity and tamper-resistance by using cryptographic hashes to uniquely identify content, aligning with DWeb principles.

Experimental results

Research questions

• RQ1 How can a decentralized search engine maintain index freshness and relevance without centralized crawling?
• RQ2 What incentive mechanisms can fairly reward content providers and peer nodes for contributing to indexing and ranking in a trustless environment?
• RQ3 How can the system resist collusion attacks where multiple peers manipulate index or ranking data maliciously?
• RQ4 What economic and cryptographic safeguards are needed to prevent scraping attacks that mimic popular content to gain rewards?
• RQ5 How can decentralized search scale securely while preserving privacy and preventing censorship?

Key findings

• QueenBee is the first known initiative to implement a decentralized search engine on the Decentralized Web, replacing traditional crawling with direct content publishing incentives.
• The system achieves trustless operation through blockchain-based smart contracts that automate reward distribution and governance.
• By eliminating centralized indexing, QueenBee enhances resilience against DDoS attacks and improves resistance to censorship and search bias.
• The incentive model rewards both content creators and peer nodes based on engagement and contribution, promoting long-term ecosystem sustainability.
• The architecture supports integration with decentralized storage (e.g., IPFS) and future Internet architectures like Named Data Networking (NDN).
• The paper identifies novel attack vectors such as collusion among worker bees and scraping of popular content, highlighting the need for robust economic and cryptographic safeguards.