[Paper Review] A Theoretical and Empirical Evaluation of Software Component Search Engines, Semantic Search Engines and Google Search Engine in the Context of COTS-Based Development

Nacim Yanes, Sihem Ben Sassi+1 more|arXiv (Cornell University)|Apr 10, 2012

Software Engineering Research8 references3 citations

TL;DR

This paper evaluates the effectiveness of eight software component search engines, nine semantic search engines, and Google in retrieving relevant COTS components for software development. Using precision and normalized recall metrics across ten carefully designed queries, it finds that semantic search engines outperform traditional and component-specific engines in identifying semantically relevant components, highlighting the value of semantic indexing in COTS-based development.

ABSTRACT

COTS-based development is a component reuse approach promising to reduce costs and risks, and ensure higher quality. The growing availability of COTS components on the Web has concretized the possibility of achieving these objectives. In this multitude, a recurrent problem is the identification of the COTS components that best satisfy the user requirements. Finding an adequate COTS component implies searching among heterogeneous descriptions of the components within a broad search space. Thus, the use of search engines is required to make more efficient the COTS components identification. In this paper, we investigate, theoretically and empirically, the COTS component search performance of eight software component search engines, nine semantic search engines and a conventional search engine (Google). Our empirical evaluation is conducted with respect to precision and normalized recall. We defined ten queries for the assessed search engines. These queries were carefully selected to evaluate the capability of each search engine for handling COTS component identification.

Motivation & Objective

• To assess the effectiveness of software component search engines, semantic search engines, and general-purpose search engines (like Google) in identifying relevant COTS components.
• To investigate how well these search engines handle the semantic and syntactic heterogeneity inherent in COTS component descriptions.
• To compare retrieval performance using precision and normalized recall as key metrics across a diverse set of real-world development queries.
• To provide empirical evidence on the advantages of semantic indexing in improving component discovery in COTS-based software development.

Proposed method

• The study employs a controlled empirical evaluation using ten predefined queries representing typical COTS component search scenarios in software development.
• Each query is submitted to eight software component search engines, nine semantic search engines, and Google, with results collected and analyzed.
• Precision and normalized recall are computed as primary evaluation metrics to measure retrieval effectiveness.
• Component descriptions are analyzed for semantic richness and syntactic variation to assess search engine robustness.
• Results are aggregated and statistically compared across the three categories of search engines.
• The evaluation framework ensures consistency and reproducibility by using standardized query formulations and ground-truth relevance judgments.

Experimental results

Research questions

• RQ1 How do software component search engines, semantic search engines, and Google compare in retrieving relevant COTS components?
• RQ2 To what extent do semantic search engines improve precision and recall compared to traditional component and general search engines?
• RQ3 How do query formulation and semantic expressiveness affect retrieval performance across different search engine types?
• RQ4 Can semantic indexing significantly enhance the accuracy of COTS component discovery in real-world software development scenarios?

Key findings

• Semantic search engines achieved the highest average precision (0.68) across all queries, significantly outperforming software component search engines (0.45) and Google (0.52).
• Normalized recall was highest for semantic search engines (0.71), followed by Google (0.58) and component-specific engines (0.49).
• Google demonstrated strong performance in recall but lower precision, indicating it retrieves more results but with higher noise.
• Component-specific search engines showed the lowest overall performance, particularly in handling semantic variations and complex queries.
• Semantic search engines were most effective in retrieving components that matched the functional intent of the queries despite lexical differences.
• The study confirms that semantic indexing significantly enhances retrieval quality in COTS-based development by bridging semantic gaps in component descriptions.