[Paper Review] Generating Correct Answers for Progressive Matrices Intelligence Tests

Niv Pekar, Yaniv Benny+1 more|arXiv (Cornell University)|Nov 1, 2020

Neural Networks and Applications15 references4 citations

TL;DR

This paper introduces a novel neural network for generating correct answers in Raven's Progressive Matrices (RPM) tests without relying on multiple-choice options—a significantly harder task than selection-based methods. By combining multi-pathway learning, dynamic variational losses, and a selective perceptual loss with hierarchical encoding, the model generates semantically accurate images that match ground truth patterns and achieve recognition performance competitive with state-of-the-art methods, even when pixel-level similarity is low.

ABSTRACT

Raven's Progressive Matrices are multiple-choice intelligence tests, where one tries to complete the missing location in a $3 imes 3$ grid of abstract images. Previous attempts to address this test have focused solely on selecting the right answer out of the multiple choices. In this work, we focus, instead, on generating a correct answer given the grid, without seeing the choices, which is a harder task, by definition. The proposed neural model combines multiple advances in generative models, including employing multiple pathways through the same network, using the reparameterization trick along two pathways to make their encoding compatible, a dynamic application of variational losses, and a complex perceptual loss that is coupled with a selective backpropagation procedure. Our algorithm is able not only to generate a set of plausible answers, but also to be competitive to the state of the art methods in multiple-choice tests.

Motivation & Objective

• To address the challenge of generating correct answers in Raven's Progressive Matrices (RPM) tests without access to multiple-choice options, which requires deeper understanding than selection-based methods.
• To develop a generative model that synthesizes plausible answers by learning underlying patterns in 3×3 grids of abstract images.
• To improve generalization and semantic reasoning in visual reasoning tasks by introducing a dynamic variational loss and selective backpropagation.
• To evaluate whether generated answers can be recognized as correct by state-of-the-art classifiers, demonstrating the model's semantic understanding.
• To validate the method on both the Procedurally Generated Matrices (PGM) and RAVEN-FAIR datasets, showing robustness across different rule distributions.

Proposed method

• The model uses three parallel pathways: reconstruction, recognition, and generation, with shared encoder networks to unify representations.
• A dynamic variational loss is applied to the reconstruction and generation paths, adapting randomness based on input context to avoid detrimental uniform noise.
• The reparameterization trick is applied selectively across two pathways to ensure compatibility between latent distributions in the reconstruction and generation tasks.
• A hierarchical perceptual loss is used, comparing generated images to ground truth and distractors via feature embeddings from a pre-trained network.
• Selective backpropagation ensures gradients update only the generation pathway, preserving the integrity of the perceptual loss networks.
• The recognition pathway acts as an auxiliary classifier, trained without the relational module on the third row and column, to improve semantic alignment.

Experimental results

Research questions

• RQ1 Can a neural network generate a correct RPM answer from scratch, without access to multiple-choice options, while maintaining semantic correctness?
• RQ2 How can variational autoencoding be adapted to support partial randomness in image generation for complex visual reasoning tasks?
• RQ3 To what extent does a perceptual loss with hierarchical encoding improve the quality and correctness of generated RPM answers?
• RQ4 Can a generative model achieve recognition accuracy comparable to state-of-the-art selection-based models when evaluated on the same classifier?
• RQ5 Does the model generalize to out-of-distribution rule sets and bias-free datasets like RAVEN-FAIR?

Key findings

• On the PGM dataset, 70.1% of generated answers were judged correct by human evaluators, compared to only 6.4% for random choice images, demonstrating strong semantic alignment.
• In a human evaluation, 63.3% of generated images were judged correct, compared to 72.2% for the ground truth, indicating high perceptual and semantic plausibility.
• The auxiliary recognition classifier achieved 60.8% accuracy on RAVEN-FAIR, second only to MRNet (86.8%), showing strong performance despite architectural constraints.
• On the RAVEN-FAIR dataset, the generation accuracy was 60.7%, close to the 69.5% accuracy of the VAE-reconstructed ground truth image, and significantly higher than the 8.9% baseline for random generation.
• In the out-of-distribution 'interpolation' regime of PGM, the model achieved 61.7% generation accuracy, approaching the 68.1% of MRNet in the recognition task.
• The model successfully captured most underlying rules in RPM problems, with minimal neglect of specific rule types, indicating robust pattern learning.