[Paper Review] CSRS: Code Search with Relevance Matching and Semantic Matching

Yi Cheng, Li Kuang|arXiv (Cornell University)|Mar 15, 2022

Software Engineering Research4 citations

TL;DR

This paper proposes CSRS, a code search model that jointly leverages neural information retrieval for lexical relevance matching and co-attention-based semantic matching to improve code retrieval performance. By combining n-gram embedding via multi-size convolutional kernels, a relevance matching module, and a semantic matching module, CSRS achieves an MRR of 0.614, outperforming state-of-the-art models DeepCS and CARLCS-CNN by 33.77% and 18.53% respectively on a large-scale code search dataset.

ABSTRACT

Developers often search and reuse existing code snippets in the process of software development. Code search aims to retrieve relevant code snippets from a codebase according to natural language queries entered by the developer. Up to now, researchers have already proposed information retrieval (IR) based methods and deep learning (DL) based methods. The IR-based methods focus on keyword matching, that is to rank codes by relevance between queries and code snippets, while DL-based methods focus on capturing the semantic correlations. However, the existing methods do not consider capturing two matching signals simultaneously. Therefore, in this paper, we propose CSRS, a code search model with relevance matching and semantic matching. CSRS comprises (1) an embedding module containing convolution kernels of different sizes which can extract n-gram embeddings of queries and codes, (2) a relevance matching module that measures lexical matching signals, and (3) a co-attention based semantic matching module to capture the semantic correlation. We train and evaluate CSRS on a dataset with 18.22M and 10k code snippets. The experimental results demonstrate that CSRS achieves an MRR of 0.614, which outperforms two state-of-the-art models DeepCS and CARLCS-CNN by 33.77% and 18.53% respectively. In addition, we also conducted several experiments to prove the effectiveness of each component of CSRS.

Motivation & Objective

• To address the limitation of existing code search methods that focus only on either keyword matching or semantic understanding, but not both simultaneously.
• To design a unified deep learning model that captures both lexical-level relevance and semantic-level correlation between natural language queries and code snippets.
• To improve code retrieval performance by integrating neural information retrieval techniques with attention-based semantic matching in a single architecture.
• To empirically validate the effectiveness of each component in the proposed model through ablation studies.

Proposed method

• The model uses a CNN-based embedding module with multiple kernel sizes (1, 2, 3) to extract n-gram embeddings from both queries and code snippets.
• A neural IR-based relevance matching module computes interaction scores between query and code embeddings using learned attention over n-gram features to capture lexical overlap.
• A co-attention mechanism is employed in the semantic matching module to dynamically align and attend to relevant parts of the query and code, capturing deeper semantic correlations.
• The relevance and semantic matching scores are fused and used to compute a final relevance score for ranking code snippets.
• The model is trained end-to-end on a large-scale dataset of 18.22M training and 10k test query-code pairs from open-source Java projects.
• Model components are evaluated via ablation studies to analyze the contribution of each module and feature type (e.g., method name, API sequence, code tokens).

Experimental results

Research questions

• RQ1 Can combining lexical relevance matching and semantic matching improve code search performance beyond current state-of-the-art models?
• RQ2 How effective is the proposed relevance matching module in capturing keyword-level similarities between queries and code snippets?
• RQ3 What is the contribution of different input features (e.g., method names, API sequences, code tokens) to the overall model performance?
• RQ4 How do different convolution kernel sizes in the embedding module affect the model's retrieval effectiveness?

Key findings

• CSRS achieves an MRR of 0.614, significantly outperforming DeepCS (MRR 0.459) and CARLCS-CNN (MRR 0.518) by 33.77% and 18.53% respectively.
• The full model with all three input features (method name, API sequence, code tokens) achieves the highest performance, with method name being the most influential individual feature.
• Using three different convolution kernel sizes (1, 2, 3) in the embedding module yields the best performance, with kernel size 1 contributing the most and the combination being optimal.
• Ablation studies confirm that both the relevance matching and semantic matching modules are essential, and removing either leads to significant performance drops.
• Qualitative examples show that CSRS successfully retrieves relevant code snippets based on both lexical overlap (e.g., 'null', 'index') and semantic understanding (e.g., 'out of range' meaning).
• The model demonstrates robustness in handling diverse queries, including those relying on semantic equivalence rather than exact keyword matches.