[Paper Review] Sluice networks: Learning what to share between loosely related tasks.

Sebastian Ruder, Joachim Bingel+2 more|arXiv (Cornell University)|May 23, 2017

Domain Adaptation and Few-Shot Learning25 references117 citations

TL;DR

Sluice Networks introduce a trainable parameterization that dynamically controls which parts of deep neural networks are shared across loosely related tasks, enabling flexible hard or soft sharing across layers, subspaces, and skip connections. The framework achieves up to 15% average error reduction across seven NLP domains using OntoNotes 5.0, outperforming standard multi-task learning approaches.

ABSTRACT

Multi-task learning is partly motivated by the observation that humans bring to bear what they know about related problems when solving new ones. Similarly, deep neural networks can profit from related tasks by sharing parameters with other networks. However, humans do not consciously decide to transfer knowledge between tasks (and are typically not aware of the transfer). In machine learning, it is hard to estimate if sharing will lead to improvements; especially if tasks are only loosely related. To overcome this, we introduce Sluice Networks, a general framework for multi-task learning where trainable parameters control the amount of sharing -- including which parts of the models to share. Our framework goes beyond and generalizes over previous proposals in enabling hard or soft sharing of all combinations of subspaces, layers, and skip connections. We perform experiments on three task pairs from natural language processing, and across seven different domains, using data from OntoNotes 5.0, and achieve up to 15% average error reductions over common approaches to multi-task learning. We analyze when the architecture is particularly helpful, as well as its ability to fit noise. We show that a) label entropy is predictive of gains in sluice networks, confirming findings for hard parameter sharing, and b) while sluice networks easily fit noise, they are robust across domains in practice.

Motivation & Objective

• To address the challenge of determining when and how to share parameters in multi-task learning, especially when tasks are only loosely related.
• To develop a framework that allows automatic, trainable control over which model components (layers, subspaces, skip connections) are shared.
• To generalize beyond existing methods by enabling all combinations of hard and soft sharing across architectural components.
• To evaluate the framework's effectiveness across diverse NLP tasks and domains, particularly in low-resource or noisy settings.
• To investigate the relationship between label entropy and performance gains, and assess robustness to noise.

Proposed method

• Introduce a parameterized gating mechanism that learns which parts of the network to share across tasks, allowing both hard and soft sharing.
• Design a modular architecture where each layer or subspace has a trainable gate controlling access to shared parameters.
• Allow sharing across any combination of layers, subspaces, and skip connections, enabling fine-grained control over parameter sharing.
• Train the entire model end-to-end using standard backpropagation, with the gate parameters updated to optimize joint task performance.
• Use a differentiable relaxation of hard sharing to allow soft sharing, enabling gradient-based optimization of sharing decisions.
• Apply the framework to multiple NLP tasks using OntoNotes 5.0, training on diverse domains to assess generalization.

Experimental results

Research questions

• RQ1 Can a learnable, adaptive sharing mechanism improve performance on loosely related multi-task learning tasks in NLP?
• RQ2 How does label entropy correlate with performance gains in the proposed framework?
• RQ3 To what extent can Sluice Networks generalize across different domains and handle noisy data?
• RQ4 Does the ability to control sharing at the layer, subspace, and skip connection level lead to better performance than fixed sharing strategies?
• RQ5 How robust are the learned sharing patterns to label noise and domain shift?

Key findings

• Sluice Networks achieve up to 15% average error reduction over standard multi-task learning baselines across seven different NLP domains using OntoNotes 5.0.
• Label entropy is predictive of performance gains, confirming that tasks with higher uncertainty benefit more from shared representations.
• The model demonstrates strong robustness across domains despite its ability to fit noise, indicating practical utility in real-world settings.
• The framework enables effective learning of which architectural components to share, outperforming fixed sharing strategies.
• The ability to learn hard or soft sharing across layers, subspaces, and skip connections leads to improved generalization and performance.
• The method maintains high performance even when task relationships are weak or ambiguous, suggesting effective inductive bias learning.