[Paper Review] Measuring Arithmetic Extrapolation Performance

Andreas Nygaard Madsen, Alexander Rosenberg Johansen|arXiv (Cornell University)|Oct 4, 2019

Model Reduction and Neural Networks10 references9 citations

TL;DR

This paper proposes a success-criterion based on a tight MSE threshold to evaluate arithmetic extrapolation performance in Neural Arithmetic Logic Units (NALUs), enabling reliable measurement of convergence success rate, convergence speed, and parameter sparsity across 4,800 experiments. It finds that NALUs, especially for multiplication and division, struggle with consistent convergence due to high sensitivity to weight initialization.

ABSTRACT

The Neural Arithmetic Logic Unit (NALU) is a neural network layer that can learn exact arithmetic operations between the elements of a hidden state. The goal of NALU is to learn perfect extrapolation, which requires learning the exact underlying logic of an unknown arithmetic problem. Evaluating the performance of the NALU is non-trivial as one arithmetic problem might have many solutions. As a consequence, single-instance MSE has been used to evaluate and compare performance between models. However, it can be hard to interpret what magnitude of MSE represents a correct solution and models sensitivity to initialization. We propose using a success-criterion to measure if and when a model converges. Using a success-criterion we can summarize success-rate over many initialization seeds and calculate confidence intervals. We contribute a generalized version of the previous arithmetic benchmark to measure models sensitivity under different conditions. This is, to our knowledge, the first extensive evaluation with respect to convergence of the NALU and its sub-units. Using a success-criterion to summarize 4800 experiments we find that consistently learning arithmetic extrapolation is challenging, in particular for multiplication.

Motivation & Objective

• To address the lack of standardized, interpretable metrics for evaluating arithmetic extrapolation in neural networks.
• To measure how often NALUs successfully learn the underlying logic of arithmetic operations across multiple random weight initializations.
• To quantify convergence speed and parameter sparsity for NALU and its sub-units (NACs) under varying task conditions.
• To provide a generalized benchmark for assessing model robustness and reliability in arithmetic extrapolation tasks.

Proposed method

• Define a success-criterion using an MSE threshold of 10^-5 relative to a nearly-perfect solution, simulating exact match for regression tasks.
• Use binomial confidence intervals to summarize success rates across 50–100 random initialization seeds, modeling convergence time with a gamma distribution.
• Measure sparsity error using max_i min(|W_i|, |1 - |W_i||), with a modified beta distribution for confidence intervals to respect bounds.
• Extend the Simple Function Learning Tasks benchmark to vary input size, subset ratio, and operation type for comprehensive evaluation.
• Train models over 1M test samples to compute reference MSE for the success-criterion, ensuring robust thresholding.
• Use maximum likelihood profiling to compute 95% confidence intervals for success rate, convergence time, and sparsity error.

Experimental results

Research questions

• RQ1 How often does the NALU successfully learn the correct arithmetic logic across different random weight initializations?
• RQ2 At what training iteration does the model typically converge to a successful solution?
• RQ3 How does model performance vary across different arithmetic operations (addition, subtraction, multiplication, division)?
• RQ4 What is the relationship between parameter sparsity and successful extrapolation in NALUs?
• RQ5 How does the generalized benchmark reveal sensitivity to hyperparameters like input size and subset ratio?

Key findings

• The NALU achieved 0% success rate on multiplication and division tasks across 100 seeds, indicating consistent failure to converge.
• For addition and subtraction, the NALU achieved 14% and 14% success rates, respectively, with high variance across initializations.
• The NAC+ sub-unit achieved 100% success rate on addition and subtraction, converging in approximately 370,000–490,000 iterations on average.
• Convergence time for successful runs ranged from 1.4M to 3.0M iterations, with the slowest convergence observed in multiplication tasks.
• Sparsity error was high (0.23–0.25) for successful NAC+ and linear models, indicating non-sparse solutions even when correct.
• The study reveals that NALUs are highly sensitive to weight initialization, with no consistent convergence across trials for multiplication and division.