[Paper Review] Neural Style Representations and the Large-Scale Classification of Artistic Style

Jeremiah Johnson|arXiv (Cornell University)|Nov 16, 2016

Aesthetic Perception and Analysis15 references3 citations

TL;DR

This paper investigates the use of neural style representations—specifically Gram matrices from a pre-trained VGG-19 network—for large-scale artistic style classification of 76,449 paintings across 70 style categories. Although a fine-tuned residual network achieved 36.99% top-1% accuracy, the best neural-style representation (using ReLU3_1 features with a random forest) reached 33.46%, demonstrating that style features from deep networks can serve as effective, albeit secondary, classifiers for artistic style.

ABSTRACT

The artistic style of a painting is a subtle aesthetic judgment used by art historians for grouping and classifying artwork. The recently introduced `neural-style' algorithm substantially succeeds in merging the perceived artistic style of one image or set of images with the perceived content of another. In light of this and other recent developments in image analysis via convolutional neural networks, we investigate the effectiveness of a `neural-style' representation for classifying the artistic style of paintings.

Motivation & Objective

• To assess whether neural style representations, derived from deep convolutional neural networks, can effectively classify artistic style in a large-scale, fine-grained dataset.
• To compare the performance of neural style representations against traditional deep learning models in artistic style classification.
• To investigate the impact of feature dimensionality and preprocessing (e.g., PCA, normalization) on the predictive power of style representations.
• To determine which layers of a pre-trained network best encode stylistic information for classification tasks.
• To evaluate the robustness of style representations in a less homogeneous, high-resolution dataset compared to prior work.

Proposed method

• Extracted Gram matrices of feature activations from VGG-19 layers ReLU1_1, ReLU2_1, ReLU3_1, ReLU4_1, and ReLU5_1 for each painting.
• Reshaped the symmetric Gram matrices into 1D feature vectors of dimensions 2016, 8128, 32640, 130816, and 130816 respectively.
• Trained a single-layer linear classifier on the full concatenated style representation using Adam optimization.
• Built separate random forest classifiers on each individual Gram matrix representation to assess layer-specific performance.
• Applied PCA to reduce dimensionality while preserving 90% of variance and evaluated its impact on classification accuracy.
• Normalized input features and evaluated their effect on model performance, particularly in conjunction with dimensionality reduction.

Experimental results

Research questions

• RQ1 Can neural style representations, as defined by Gram matrices of feature activations, serve as effective classifiers for artistic style in a large-scale, fine-grained dataset?
• RQ2 How does the performance of neural style representations compare to that of a fine-tuned residual neural network on the same classification task?
• RQ3 Which specific layers of a pre-trained VGG-19 network yield the most discriminative style representations for artistic style classification?
• RQ4 What is the impact of dimensionality reduction (e.g., PCA) on the predictive accuracy of neural style representations?
• RQ5 Does feature normalization improve the performance of models trained on neural style representations?

Key findings

• The best-performing neural style representation was obtained from the ReLU3_1 layer using a random forest classifier, achieving a top-1% accuracy of 33.46%.
• The ReLU2_1 and ReLU1_1 layers yielded 28.97% and 27.84% top-1% accuracy respectively, outperforming the full-feature linear classifier (13.21% accuracy).
• The performance of neural style representations degraded significantly in deeper layers (ReLU4_1 and ReLU5_1), with accuracies dropping to 9.79% and 10.18% respectively.
• Applying PCA to preserve 90% of variance in the ReLU1_1 representation reduced random forest accuracy from 27.84% to 17%, indicating that high-dimensional structure is critical for performance.
• Normalization of input features provided no significant improvement in model accuracy, suggesting that raw Gram matrix features are already well-scaled for classification.
• Despite lower performance than the fine-tuned residual network (36.99% top-1% accuracy), neural style representations demonstrated meaningful predictive power, indicating they encode relevant stylistic information.