[Paper Review] Deep Symbolic Regression for Recurrent Sequences

Stéphane d’Ascoli, Pierre-Alexandre Kamienny+2 more|arXiv (Cornell University)|Jan 12, 2022

Evolutionary Algorithms and Applications26 citations

TL;DR

The paper trains Transformer models to infer recurrence relations from early terms of sequences (integers and floats), evaluating on OEIS and out-of-vocabulary constants to show symbolic regression benefits over purely numeric extrapolation.

ABSTRACT

Symbolic regression, i.e. predicting a function from the observation of its values, is well-known to be a challenging task. In this paper, we train Transformers to infer the function or recurrence relation underlying sequences of integers or floats, a typical task in human IQ tests which has hardly been tackled in the machine learning literature. We evaluate our integer model on a subset of OEIS sequences, and show that it outperforms built-in Mathematica functions for recurrence prediction. We also demonstrate that our float model is able to yield informative approximations of out-of-vocabulary functions and constants, e.g. $\operatorname{bessel0}(x)\approx \frac{\sin(x)+\cos(x)}{\sqrt{πx}}$ and $1.644934\approx π^2/6$. An interactive demonstration of our models is provided at https://symbolicregression.metademolab.com.

Motivation & Objective

• Motivate the challenge of inferring recurrences from sequences, including integers and floats.
• Develop Transformer-based models to perform symbolic and numeric regression for recurrent expressions.
• Create synthetic data generation pipelines with controllable recurrence depth and operator sets.
• Evaluate in-domain and out-of-domain generalization against established baselines.
• Demonstrate the ability to approximate out-of-vocabulary constants and functions with learned vocabularies.

Proposed method

• Use an 8-layer Transformer encoder-decoder with 512-dim embeddings to map sequences to recurrence expressions or next-term sequences.
• Represent sequences and candidate expressions as token sequences using base-b integer encoding and base-10 float tokens, with prefix (Polish) encoding for trees.
• Train with cross-entropy loss and Adam optimizer, with a warmup schedule and inverse-square-root decay.
• Generate training data by randomly constructing unary-binary operator trees, sampling leaves (constants, indices, or previous terms), and producing sequences of length up to l.
• Evaluate hypotheses with beam search (size 10) and re-rank by how well predicted recurrences reproduce initial terms.
• Compare symbolic regression (recurrence inference) with numeric regression (direct next-term prediction) and test on in-domain and out-of-domain data.

Experimental results

Research questions

• RQ1 Can transformers learn to infer recurrence relations from observed sequence terms in both integer and float settings?
• RQ2 Do symbolic models extrapolate with higher precision than numeric models on recurrent sequences?
• RQ3 How well do learned models generalize to out-of-domain sequences such as certain OEIS entries or out-of-vocabulary constants and operators?
• RQ4 What are the limitations and failure modes of symbolic recurrence inference in this setting?
• RQ5 How does the model handle in-domain versus out-of-domain sequences and what are the factors that affect performance (operator count, recurrence degree, sequence length)?

Key findings

• The symbolic model achieves high in-domain accuracy and often extrapolates further with higher precision than the numeric model.
• On integer sequences, the symbolic model reaches 92.7% accuracy for n_op<=5 and 83.6% for numeric predictions in the specified setup, while the float model achieves 74.2% symbolic and 45.6% numeric under n_op<=5.
• The integer OEIS subset evaluation shows the symbolic model can retrieve valid recurrence relations for several sequences, outperforming Mathematica’s FindSequenceFunction and FindLinearRecurrence in the reported settings.
• The float model demonstrates the ability to approximate out-of-vocabulary constants and functions, producing asymptotic or exact expressions (e.g., Bessel j0 asymptotics) and close approximations to constants such as pi^2/6.
• Out-of-domain experiments indicate the model can generalize to unseen recurrences and out-of-vocabulary tokens, though performance degrades with increased difficulty and operator variety.
• Ablation analyses show accuracy drops with more operators, higher recurrence degree, and shorter input lengths, with symbolic models showing more robustness to longer prediction horizons.