[Paper Review] MuFuRU: The Multi-Function Recurrent Unit

Dirk Weissenborn, Tim Rocktäschel|arXiv (Cornell University)|Jun 9, 2016

Topic Modeling17 references3 citations

TL;DR

MuFuRU introduces a novel recurrent neural network unit that learns to select from a set of arbitrary differentiable composition operations (e.g., element-wise max, min, absolute difference) to update its hidden state, with operation choice dependent on both input and previous state. It outperforms standard GRUs on language modeling and sentiment analysis and generalizes better than GRUs on logical reasoning tasks, even with smaller memory sizes and no overfitting.

ABSTRACT

Recurrent neural networks such as the GRU and LSTM found wide adoption in natural language processing and achieve state-of-the-art results for many tasks. These models are characterized by a memory state that can be written to and read from by applying gated composition operations to the current input and the previous state. However, they only cover a small subset of potentially useful compositions. We propose Multi-Function Recurrent Units (MuFuRUs) that allow for arbitrary differentiable functions as composition operations. Furthermore, MuFuRUs allow for an input- and state-dependent choice of these composition operations that is learned. Our experiments demonstrate that the additional functionality helps in different sequence modeling tasks, including the evaluation of propositional logic formulae, language modeling and sentiment analysis.

Motivation & Objective

• To address the limitation of existing RNNs like GRUs and LSTMs, which are restricted to a small set of fixed composition operations (e.g., gated addition or replacement), by enabling arbitrary differentiable functions as composition operations.
• To allow the network to learn input- and state-dependent selection of these operations end-to-end from task data, increasing architectural adaptability.
• To demonstrate that this flexibility leads to improved generalization and performance on sequence modeling tasks beyond what standard RNN units achieve.
• To validate the model's ability to learn complex, task-specific behaviors—such as evaluating propositional logic—without overfitting, even with small memory sizes.

Proposed method

• MuFuRU defines a set of differentiable composition operations (e.g., max, min, absolute difference, multiplication) that can be applied to the current input features and the previous hidden state.
• At each time step, an operation controller computes normalized attention-like weights over the available operations using a feedforward network applied to the concatenation of the current input and previous hidden state.
• The final hidden state is computed as a weighted sum of the outputs from each composition operation, where the weights are determined by the learned operation controller.
• The operation controller is trained end-to-end via backpropagation, enabling dynamic, context-dependent selection of the most appropriate composition function at each step.
• The architecture generalizes GRUs and LSTMs by allowing a broader class of functions and learning the selection mechanism, rather than fixing it via hard-coded gates.
• The method supports integration into any RNN framework and allows plug-in of new task-specific differentiable functions without architectural changes.

Experimental results

Research questions

• RQ1 Can a recurrent unit that learns to select from a diverse set of differentiable composition operations generalize better than standard GRUs on sequence modeling tasks?
• RQ2 Does the ability to dynamically choose operations based on input and state lead to improved performance on tasks requiring complex state transformations, such as logical reasoning?
• RQ3 Can MuFuRU learn to emulate logical operations (e.g., AND, OR, NOT) in propositional logic evaluation with smaller memory capacity and without overfitting?
• RQ4 How does the performance of MuFuRU compare to standard GRUs and state-of-the-art models on language modeling and sentiment analysis, especially without hyperparameter tuning?
• RQ5 To what extent does the learned operation selection reflect meaningful, interpretable behavior rather than random or redundant function use?

Key findings

• MuFuRU achieved a test perplexity of 119.7 on the Penn Treebank language modeling task, outperforming the GRU baseline, which had a perplexity of 123.0.
• On the propositional logic evaluation task, MuFuRU achieved 87.6% accuracy on unseen formulae with 11–20 gates using only 8 hidden units, while the GRU required significantly larger dimensions and overfit quickly.
• The MuFuRU demonstrated no overfitting even with large hidden dimensions on the logic task, indicating it learned a generalizable, operation-specific strategy rather than memorizing patterns.
• In sentiment analysis on the Sentiment Treebank, MuFuRU outperformed the GRU baseline and approached state-of-the-art results without architectural modifications or hyperparameter tuning.
• Visualization of operation weights showed that MuFuRU made sparse, context-dependent use of operations, with learned behavior tailored to input and state—e.g., using multiplication heavily for OR operations despite counterintuitive expectations.
• The model’s ability to learn and apply diverse composition functions enabled it to generalize across sequence lengths and tasks, suggesting it captures more flexible and task-adaptive representations than standard RNN units.