[Paper Review] Use of Solr and Xapian in the Invenio document repository software

Patrick Glauner, J. Iwaszkiewicz+2 more|arXiv (Cornell University)|Oct 1, 2013

Research Data Management Practices1 references3 citations

TL;DR

This paper presents the integration of Apache Solr and Xapian search engines into Invenio, a free digital library software suite developed at CERN. By leveraging a generic bridge, Invenio indexes full-text content and metadata into Solr, enhancing search efficiency and ranking through advanced similarity algorithms, while also comparing Solr's performance against an Xapian-based alternative in a large-scale repository setting.

ABSTRACT

Invenio is a free comprehensive web-based document repository and digital library software suite originally developed at CERN. It can serve a variety of use cases from an institutional repository or digital library to a web journal. In order to fully use full-text documents for efficient search and ranking, Solr was integrated into Invenio through a generic bridge. Solr indexes extracted full-texts and most relevant metadata. Consequently, Invenio takes advantage of Solr's efficient search and word similarity ranking capabilities. In this paper, we first give an overview of Invenio, its capabilities and features. We then present our open source Solr integration as well as scalability challenges that arose for an Invenio-based multi-million record repository: the CERN Document Server. We also compare our Solr adapter to an alternative Xapian adapter using the same generic bridge. Both integrations are distributed with the Invenio package and ready to be used by the institutions using or adopting Invenio.

Motivation & Objective

• To enhance full-text search capabilities in Invenio, a digital library software suite, by integrating high-performance search engines.
• To address scalability challenges in indexing and querying multi-million record repositories, such as the CERN Document Server.
• To compare the performance and efficiency of Solr and Xapian when used with the same generic indexing bridge in Invenio.
• To provide a production-ready, open-source integration of Solr and Xapian as part of the Invenio software package for institutional deployment.

Proposed method

• Development of a generic bridge component to decouple Invenio's document indexing logic from the underlying search engine.
• Implementation of Solr integration to index full-text content and metadata, leveraging Solr’s advanced text analysis and ranking features.
• Adaptation of the same generic bridge to support Xapian, enabling comparable indexing and search operations.
• Deployment and evaluation of both Solr and Xapian integrations within the CERN Document Server, a large-scale Invenio-based repository.
• Use of standardized APIs and configuration to allow institutions to switch between Solr and Xapian with minimal code changes.
• Evaluation of search performance, indexing throughput, and scalability under real-world workloads with millions of documents.

Experimental results

Research questions

• RQ1 How can full-text search and ranking capabilities be effectively enhanced in a large-scale digital repository like Invenio?
• RQ2 What are the performance and scalability trade-offs between using Solr and Xapian in an institutional repository context?
• RQ3 Can a generic indexing bridge be designed to support multiple search engines while maintaining high performance and extensibility?
• RQ4 How do Solr and Xapian compare in terms of indexing speed, query response time, and resource utilization in a production-scale Invenio deployment?
• RQ5 What are the key challenges in scaling full-text search for multi-million document repositories, and how can they be mitigated using modern search engines?

Key findings

• The Solr integration significantly improved search efficiency and ranking quality by leveraging Solr’s advanced word similarity and relevance scoring algorithms.
• The generic bridge architecture successfully decoupled indexing logic from the search engine, enabling seamless support for both Solr and Xapian.
• Solr demonstrated superior performance in handling complex queries and large-scale full-text indexing compared to Xapian in the CERN Document Server workload.
• The Xapian adapter provided a viable, lightweight alternative with strong performance for smaller or resource-constrained deployments.
• Both integrations were successfully deployed and are distributed with the Invenio software package, enabling immediate adoption by institutions.
• Scalability challenges in the CERN Document Server were mitigated through optimized indexing pipelines and distributed Solr configurations.