[Paper Review] Visual Reasoning by Progressive Module Networks

Seung Wook Kim, Makarand Tapaswi+1 more|arXiv (Cornell University)|Jun 6, 2018

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes Progressive Module Networks (PMN), a differentiable, compositional framework that trains visual reasoning models by progressively building higher-level tasks from reusable, black-box lower-level modules. By learning to query and compose outputs from existing modules via a soft gating mechanism, PMN achieves improved performance across visual reasoning tasks, enhanced interpretability through explicit reasoning traces, and data efficiency by reusing pre-trained skills without catastrophic forgetting.

ABSTRACT

Humans learn to solve tasks of increasing complexity by building on top of previously acquired knowledge. Typically, there exists a natural progression in the tasks that we learn - most do not require completely independent solutions, but can be broken down into simpler subtasks. We propose to represent a solver for each task as a neural module that calls existing modules (solvers for simpler tasks) in a functional program-like manner. Lower modules are a black box to the calling module, and communicate only via a query and an output. Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output. Our model effectively combines previous skill-sets, does not suffer from forgetting, and is fully differentiable. We test our model in learning a set of visual reasoning tasks, and demonstrate improved performances in all tasks by learning progressively. By evaluating the reasoning process using human judges, we show that our model is more interpretable than an attention-based baseline.

Motivation & Objective

• To address the limitations of conventional multi-task learning, where tasks are disconnected and do not build on each other’s knowledge.
• To enable neural networks to learn increasingly complex visual reasoning tasks by composing solutions from simpler, pre-trained modules.
• To improve model interpretability by making the reasoning process explicit through query and response traces of composed modules.
• To enhance data efficiency by reusing previously learned modules, avoiding catastrophic forgetting and reducing data requirements.

Proposed method

• Each task is represented as a neural module that can query and compose outputs from lower-level modules via a soft gating mechanism.
• Modules communicate only through queries and outputs, treating lower-level modules as black boxes, enabling modular composition and reuse.
• The model is fully differentiable, allowing end-to-end training of both module composition and communication parameters.
• A rule-based natural language explanation is generated from the sequence of modules used, enabling human evaluation of reasoning quality.
• The framework supports selective information flow, where parent modules learn which child modules to query based on task requirements.
• The architecture is designed to prevent catastrophic forgetting by reusing pre-trained modules and only learning inter-module communication.

Experimental results

Research questions

• RQ1 Can a neural network learn to solve complex visual reasoning tasks by composing solutions from simpler, pre-trained modules?
• RQ2 Does progressive composition of modules lead to improved performance and data efficiency compared to end-to-end training?
• RQ3 Can the reasoning process be made interpretable by exposing the sequence of module queries and responses?
• RQ4 How does the model perform under low-data regimes, and does it maintain performance by reusing pre-trained components?
• RQ5 Can the model outperform attention-based baselines in both accuracy and interpretability, especially on complex reasoning tasks?

Key findings

• PMN outperforms non-compositional baselines on all evaluated visual reasoning tasks, including VQA, counting, captioning, and relationship reasoning.
• Human evaluators rated PMN’s reasoning explanations significantly higher than the baseline’s, with a 1.38-point advantage on correct answers and a 0.74-point advantage on correct PMN answers versus wrong baseline answers.
• In low-data regimes, PMN achieved a maximum absolute accuracy gain of 4.01% when using only 10% of the training data, demonstrating strong data efficiency.
• The performance gain from module reuse remained stable at around 2% when using 25% or more of the training data, indicating consistent benefit from knowledge transfer.
• PMN’s reasoning process was found to be more robust to errors: even when the final answer was incorrect, partial correctness in reasoning steps led to higher human ratings than incorrect reasoning in the baseline.
• The model’s ability to compose modules enabled it to solve complex questions by breaking them into subtasks, such as first detecting objects and then reasoning about spatial relationships or attributes.