[Paper Review] TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts

Michihiro Yasunaga, John Lafferty|arXiv (Cornell University)|Feb 16, 2019

Mathematics, Computing, and Information Processing40 references4 citations

TL;DR

TopicEq proposes a joint topic and equation model that generates scientific text and mathematical equations using shared latent topics, with equations produced by an RNN conditioned on topic activations. The model significantly outperforms baseline topic and equation models, enabling novel applications like topic-aware equation generation and symbol-word alignment.

ABSTRACT

Scientific documents rely on both mathematics and text to communicate ideas. Inspired by the topical correspondence between mathematical equations and word contexts observed in scientific texts, we propose a novel topic model that jointly generates mathematical equations and their surrounding text (TopicEq). Using an extension of the correlated topic model, the context is generated from a mixture of latent topics, and the equation is generated by an RNN that depends on the latent topic activations. To experiment with this model, we create a corpus of 400K equation-context pairs extracted from a range of scientific articles from arXiv, and fit the model using a variational autoencoder approach. Experimental results show that this joint model significantly outperforms existing topic models and equation models for scientific texts. Moreover, we qualitatively show that the model effectively captures the relationship between topics and mathematics, enabling novel applications such as topic-aware equation generation, equation topic inference, and topic-aware alignment of mathematical symbols and words.

Motivation & Objective

• To address the gap in jointly modeling topics and mathematical equations in scientific texts, where current methods treat them separately.
• To improve topic modeling in scientific documents by incorporating syntactic and semantic information from equations via an RNN.
• To enhance equation generation by leveraging contextual word information from surrounding text and topic structure.
• To enable interpretable, topic-aware associations between mathematical symbols and their natural language descriptions.
• To develop an unsupervised framework that supports novel applications without manual annotations.

Proposed method

• Extends the correlated topic model to jointly generate text context and mathematical equations using shared latent topic proportions.
• Uses an RNN to generate equations as sequences of LaTeX symbols, conditioned on the topic activation vector.
• Applies neural variational inference via a variational autoencoder to perform approximate posterior inference on the large-scale ContextEq-400K corpus.
• Introduces a topic-aware alignment model that learns symbol-word associations using a topic-dependent alignment matrix parameterized by topic vectors.
• Employs a multi-layered alignment matrix $ A(\theta) = W_a \cdot \text{diag}(W_b\theta) \cdot W_c $ to model context-dependent symbol meanings.
• Trains the model end-to-end on 400K equation-context pairs extracted from arXiv articles, using LATEX source for accurate parsing.

Experimental results

Research questions

• RQ1 Can joint modeling of topics and equations improve the quality of topic modeling in scientific texts compared to treating equations as bags of tokens?
• RQ2 Can incorporating contextual text and topic structure enhance the grammaticality and relevance of generated mathematical equations?
• RQ3 To what extent can the model learn topic-aware associations between mathematical symbols and their natural language descriptions?
• RQ4 How does the joint model compare to standalone topic models and equation models in terms of perplexity and coherence?
• RQ5 Can the model support novel applications such as topic-aware equation generation and symbol meaning inference without labeled data?

Key findings

• The joint TopicEq model reduces test perplexity by over 33% compared to a baseline alignment model that ignores topic context, demonstrating the value of topic-aware symbol-word alignment.
• Topic coherence scores for the joint model reach 0.088 with 50 topics and 0.087 with 100 topics, showing moderate but consistent improvement over the baseline topic model (0.085/0.084).
• The model successfully learns contextually appropriate meanings for symbols: for example, $E$ is associated with "expectation" in probability, "electric field" in quantum physics, and "edge" in graph theory.
• The baseline model without topic context associates $E$ primarily with "energy" due to its high frequency across domains, highlighting the benefit of topic-aware modeling.
• Topic-aware equation generation is feasible and effective, with generated equations reflecting the semantic and syntactic patterns of their associated topics.
• The model enables unsupervised equation topic inference and symbol-word alignment, offering practical utility for readers encountering unfamiliar mathematical content.