[Paper Review] Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retrieval in Asymmetric Texts

Pankaj Gupta, Bernt Andrassy+1 more|arXiv (Cornell University)|Jul 8, 2018

Topic Modeling4 citations

TL;DR

This paper proposes a Replicated Siamese LSTM architecture to improve semantic similarity learning and retrieval in industrial ticketing systems with asymmetric text pairs—short queries (subject, description) versus longer historical tickets (subject, description, solution). By leveraging multi-level (subject, description, solution) and multi-channel (distributed embeddings, LSTM, topic vectors) representations, the model achieves a 22% gain in Accuracy@10 over unsupervised baselines and 7% over supervised baselines, demonstrating significant improvement in real-world similarity retrieval tasks.

ABSTRACT

The goal of our industrial ticketing system is to retrieve a relevant solution for an input query, by matching with historical tickets stored in knowledge base. A query is comprised of subject and description, while a historical ticket consists of subject, description and solution. To retrieve a relevant solution, we use textual similarity paradigm to learn similarity in the query and historical tickets. The task is challenging due to significant term mismatch in the query and ticket pairs of asymmetric lengths, where subject is a short text but description and solution are multi-sentence texts. We present a novel Replicated Siamese LSTM model to learn similarity in asymmetric text pairs, that gives 22% and 7% gain (Accuracy@10) for retrieval task, respectively over unsupervised and supervised baselines. We also show that the topic and distributed semantic features for short and long texts improved both similarity learning and retrieval.

Motivation & Objective

• Address the challenge of semantic similarity learning between short queries and long historical tickets in industrial ticketing systems, where term mismatch and structural asymmetry hinder retrieval.
• Improve information retrieval performance by modeling multi-level and cross-level semantic similarities across components (subject, description, solution) of query-ticket pairs.
• Enhance retrieval accuracy using distributed word embeddings, LSTM-encoded representations, and neural topic modeling as complementary input channels.
• Demonstrate the effectiveness of a novel Replicated Siamese LSTM architecture in real-world industrial applications with limited labeled data.
• Overcome limitations of existing Siamese networks, which typically handle symmetric sentence pairs, by extending them to asymmetric, multi-component text pairs.

Proposed method

• Propose a Replicated Siamese LSTM architecture where dual branches process query (SUB1+DESC1) and historical ticket (SUB2+DESC2+SOL2) components with tied weights to learn shared representations.
• Integrate multi-channel inputs: word embeddings (SumEMB), LSTM-encoded hidden states (LSTM), and topic vectors (T) to enrich semantic representation.
• Define a multi-channel Manhattan metric loss function that computes pairwise similarities across all component pairs (e.g., SUB1 vs. SUB2, SUB1 vs. DESC2, etc.) using weighted L1 distances.
• Formulate the objective function as an exponential of negative weighted L1 distances between representations: $ g = \exp\big(-\sum_{p,q} V_{\{p,q\}} (W_h\|h_p - h_q\|_1 + W_E\|E_p - E_q\|_1 + W_T\|T_p - T_q\|_1) \big) $.
• Use DocNADE to learn topic vectors from documents, enabling topic-aware semantic modeling in long texts like descriptions and solutions.
• Train the model end-to-end using contrastive loss to optimize similarity scores between relevant query-ticket pairs.

Experimental results

Research questions

• RQ1 Can a Replicated Siamese LSTM architecture effectively model semantic similarity across asymmetric text pairs with varying lengths and structures?
• RQ2 Does incorporating multi-level (subject, description, solution) and multi-channel (word embeddings, LSTM, topic vectors) representations improve similarity learning in industrial ticketing systems?
• RQ3 How does the proposed model compare to unsupervised and supervised baselines in terms of retrieval performance on real-world ticket data?
• RQ4 To what extent do topic modeling and distributed semantic features enhance similarity learning in short and long text components?
• RQ5 Can the model generalize to real-world industrial applications with limited labeled data and significant term mismatch?

Key findings

• The proposed Replicated Siamese LSTM model achieves a 22% absolute improvement in Accuracy@10 over unsupervised baselines for retrieval in the industrial ticketing system.
• The model achieves a 7% absolute improvement in Accuracy@10 over supervised baselines, demonstrating strong performance even with limited labeled data.
• The integration of topic vectors and distributed semantic features significantly enhances similarity learning, especially in long-form descriptions and solutions.
• Cross-level similarities (e.g., query subject to ticket solution) contribute meaningfully to retrieval performance, validating the multi-level modeling approach.
• The model outperforms standard Siamese networks by explicitly modeling asymmetric text pairs across multiple components and representation channels.
• The architecture generalizes well across industrial domains and reduces reliance on hand-crafted linguistic features, making it broadly applicable.