[Paper Review] Query Understanding for Natural Language Enterprise Search

Francisco Borges, Georgios Balikas+6 more|arXiv (Cornell University)|Dec 11, 2020

Topic Modeling29 references4 citations

TL;DR

This paper presents a production-ready Natural Language Search (NLS) system for a major CRM platform that combines deep learning-based Named Entity Recognition (NER) with rule-based expert systems and a grammar-based query suggestion engine. The hybrid architecture significantly reduces user effort—user studies show NLS saves over 50% time compared to navigational search—by enabling natural language queries to dynamically retrieve records, with key innovations in synthetic training data generation and robust model lifecycle management for enterprise constraints.

ABSTRACT

Natural Language Search (NLS) extends the capabilities of search engines that perform keyword search allowing users to issue queries in a more "natural" language. The engine tries to understand the meaning of the queries and to map the query words to the symbols it supports like Persons, Organizations, Time Expressions etc.. It, then, retrieves the information that satisfies the user's need in different forms like an answer, a record or a list of records. We present an NLS system we implemented as part of the Search service of a major CRM platform. The system is currently in production serving thousands of customers. Our user studies showed that creating dynamic reports with NLS saved more than 50% of our user's time compared to achieving the same result with navigational search. We describe the architecture of the system, the particularities of the CRM domain as well as how they have influenced our design decisions. Among several submodules of the system we detail the role of a Deep Learning Named Entity Recognizer. The paper concludes with discussion over the lessons learned while developing this product.

Motivation & Objective

• To design and deploy a scalable, production-grade Natural Language Search (NLS) system within a complex, privacy-sensitive CRM environment.
• To address the challenge of limited, domain-specific training data for NER in enterprise settings with high customization and data privacy constraints.
• To improve user efficiency by enabling natural language queries to dynamically retrieve records, reducing reliance on navigational or report-based workflows.
• To develop a robust, hybrid architecture that combines deep learning with rule-based systems for better generalization and reliability across diverse customer configurations.
• To establish best practices for model lifecycle management, evaluation, and instrumentation in real-world NLP systems under evolving user demands.

Proposed method

• Employed a hybrid architecture with two parallel tagging pipelines: one combining deep learning NER and rule-based expert systems for generalization, and another using a context-free grammar (Query Suggestions Grammar) for predictable query suggestion execution.
• Generated synthetic, realistic training data using a Probabilistic Context-Free Grammar (PCFG) to overcome scarcity of real-world enterprise query annotations.
• Trained domain-specific word embeddings tailored to CRM-specific query patterns, moving beyond general pre-trained embeddings for improved entity recognition.
• Implemented a multi-stage quality pipeline with versioned model gating, metadata-driven algorithms, and A/B testing to ensure production stability and measurable improvement.
• Integrated expert systems for simple, high-coverage tasks (e.g., picklist values, boolean filters) to enhance robustness and reduce pressure on deep learning models.
• Used online monitoring and instrumentation to track system behavior and failures, guiding iterative improvements and model updates.

Experimental results

Research questions

• RQ1 How can a deep learning-based NER model be effectively adapted to enterprise CRM systems with limited, high-variability training data?
• RQ2 What architectural patterns best balance the flexibility of deep learning with the reliability of rule-based systems in a multi-tenant, privacy-constrained environment?
• RQ3 To what extent can synthetic, PCFG-generated training data improve NER performance when real annotated data is scarce?
• RQ4 How can model lifecycle management and evaluation pipelines be designed to support continuous improvement in a production NLP system with evolving user needs?
• RQ5 What role do hybrid systems—combining neural and symbolic components—play in achieving robust, scalable enterprise search?

Key findings

• The NLS system reduced user time by over 50% compared to traditional navigational search when creating dynamic reports, demonstrating significant workflow acceleration.
• Synthetic data generation using a PCFG produced realistic, semantically meaningful training data that effectively improved model generalization despite limited real-world annotations.
• The hybrid architecture—combining deep learning NER with rule-based expert systems and a grammar-based query suggestion engine—provided better robustness and coverage than pure neural or symbolic approaches alone.
• Expert systems proved more reliable than deep learning for simple, structured tasks like recognizing picklist values or boolean filters, especially in highly customized customer environments.
• Model versioning, metadata-driven pipelines, and A/B testing were critical for maintaining quality and enabling iterative improvements in a production setting with evolving user demands.
• The system evolved from a pure deep learning NER model to a hybrid system, reflecting the need for adaptability and reliability in real-world enterprise deployments.