[Paper Review] IQ of Neural Networks

Dokhyam Hoshen, Michael Werman|arXiv (Cornell University)|Sep 29, 2017

Neural Networks and Applications14 references16 citations

TL;DR

This paper proposes a convolutional neural network (CNN) model that solves geometric pattern recognition problems from IQ tests by learning transformations such as rotation, reflection, size changes, and color shifts. Trained on synthetic data, the model achieves top 5% human performance on real IQ test questions, scoring 38 out of 40 correct, demonstrating strong generalization to real-world pattern recognition tasks.

ABSTRACT

IQ tests are an accepted method for assessing human intelligence. The tests consist of several parts that must be solved under a time constraint. Of all the tested abilities, pattern recognition has been found to have the highest correlation with general intelligence. This is primarily because pattern recognition is the ability to find order in a noisy environment, a necessary skill for intelligent agents. In this paper, we propose a convolutional neural network (CNN) model for solving geometric pattern recognition problems. The CNN receives as input multiple ordered input images and outputs the next image according to the pattern. Our CNN is able to solve problems involving rotation, reflection, color, size and shape patterns and score within the top 5% of human performance.

Motivation & Objective

• To evaluate deep learning models directly on tasks designed to measure human intelligence, specifically pattern recognition in IQ tests.
• To develop a neural network capable of learning and generalizing geometric transformations such as rotation, reflection, size, color, and shape addition.
• To benchmark machine learning performance against human performance on standardized IQ test problems, particularly Raven’s Progressive Matrices.
• To explore the limitations of CNNs in generating high-fidelity image outputs for open-ended IQ questions and assess transferability to realistic visual patterns.
• To improve adaptive testing by using model probabilities to assess question difficulty and confusion levels.

Proposed method

• Train a CNN on synthetic datasets of image sequences where each image is a transformed version of the previous one using predefined geometric operations.
• Use two evaluation modes: multiple-choice selection and direct image generation (regression) to predict the next image in the sequence.
• Apply data augmentation with transformations including rotation, scaling, reflection, shape addition, color changes, and number of objects.
• Employ an autoencoder-based architecture to encode input images and decode predicted outputs, minimizing mean squared error (MSE) between ground truth and prediction.
• Test the model on real-world IQ test questions from the National Institute for Testing and Evaluation in Israel, using a held-out test set of 40 questions.
• Evaluate robustness by adding Gaussian noise (σ = 99) to inputs and outputs to simulate real-world visual noise.

Experimental results

Research questions

• RQ1 Can a CNN learn and generalize geometric transformations such as rotation, reflection, and scaling from image sequences without explicit supervision?
• RQ2 How does the performance of a deep learning model on IQ-style pattern recognition tasks compare to human performance?
• RQ3 What are the relative difficulties of different transformation types (e.g., rotation vs. color) for both neural networks and humans?
• RQ4 To what extent can a model trained on synthetic data generalize to real IQ test questions with complex, realistic shapes?
• RQ5 Can model confidence scores (probabilities) be used to assess question difficulty and confusion levels in a way that improves adaptive testing?

Key findings

• The model achieved a 95% accuracy rate (38 out of 40 correct answers) on a real IQ test set from Israel’s National Institute for Testing and Evaluation, placing it within the top 5% of human performance.
• The average mean squared error (MSE) across all transformation types was 3.5×10⁻⁴, with reflection and rotation tasks showing the highest error (4.1×10⁻⁴ and 5.7×10⁻⁴, respectively), indicating greater learning difficulty.
• The model performed robustly under noisy conditions, with a 5% improvement in multiple-choice error rate (87% vs. 91% in clean data), suggesting resilience to visual noise.
• The model struggled to generate crisp, high-fidelity image outputs in the open-ended (regression) format when faced with complex, realistic shapes, indicating a limitation in transfer learning to naturalistic visual patterns.
• Empirical results confirm that reflection and rotation tasks are more difficult for both the model and humans, aligning with known human cognitive difficulty in IQ tests.
• The model’s performance on color and size transformation tasks was superior, with MSEs of 2.1×10⁻⁴ and 2.4×10⁻⁴, respectively, due to fewer parameters to infer compared to rotation/reflection.