[Paper Review] Deep Learning for Symbolic Mathematics

Guillaume Lample, François Charton|arXiv (Cornell University)|Dec 2, 2019

Natural Language Processing TechniquesComputer Science29 references47 citations

TL;DR

The authors show that sequence-to-sequence models can effectively perform symbolic mathematics, notably competitive with or surpassing Matlab/Mathematica on symbolic integration and solving ODEs, by representing expressions as prefix trees and generating large synthetic datasets.

ABSTRACT

Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.

Motivation & Objective

• Propose a syntax to represent mathematical problems suitable for neural seq2seq models.
• Generate large, diverse datasets for symbolic tasks including integration and ODEs.
• Train transformer-based models to predict symbolic solutions and evaluate against CAS tools.
• Demonstrate that neural models can outperform commercial systems on these tasks.

Proposed method

• Represent expressions as trees, and encode them as prefix sequences to feed into seq2seq models.
• Generate random expressions with controlled operator/leaf counts and study the combinatorial space (Catalan/Schroeder numbers).
• Create forward, backward, and integration-by-parts data-generation pipelines for integration problems, plus methods to generate solvable first- and second-order ODEs.
• Train Transformer models (8 heads, 6 layers, hidden 512) with Adam, using beam search during decoding to generate candidate solutions.
• Evaluate model-generated solutions by symbolic equivalence checks against references (and against CAS outputs when available).
• Compare performance with Mathematica, Maple, and Matlab under timeouts and various beam widths.

Experimental results

Research questions

• RQ1 Can neural seq2seq models learn to perform symbolic integration and solve differential equations from synthetic data?
• RQ2 How does data generation strategy (FWD, BWD, IBP) affect model learning and generalization?
• RQ3 To what extent can neural models outperform traditional Computer Algebra Systems on these symbolic tasks?
• RQ4 How does beam search affect accuracy for symbolic problem solving?
• RQ5 Are the generated solutions equivalent to reference solutions under symbolic equivalence?

Key findings

• The model achieves near 100% accuracy on integration across three data-generation methods with greedy decoding.
• Beam size significantly improves ODE solving: order-1 improves from 77.6% (beam 1) to 90.5% (beam 10) to 93.97% (beam 50); order-2 improves from 43.0% (beam 1) to 73.0% (beam 10) to 81.17% (beam 50).
• On a held-out test set of 500 equations, Mathematica with a 30s timeout solves 84.0% for BWD integration, while the neural model with beam 50 reaches 99.6% for integration in some configurations and 97.0% for order-2 ODEs.
• Overall, neural models significantly outperform Mathematica on integration and, with larger beams, outperform Mathematica on ODEs as well.
• The model can generate multiple equivalent but differently written solutions for a problem, demonstrating nontrivial symbolic equivalence recognition without explicit training for equivalence.