[Paper Review] Formal Analysis of Art: Proxy Learning of Visual Concepts from Style Through Language Models

Diana Kim, Ahmed Elgammal+1 more|arXiv (Cornell University)|Jan 5, 2022

Aesthetic Perception and Analysis4 citations

TL;DR

This paper proposes a proxy learning framework that quantifies visual elements in fine art paintings using style labels and language models, bypassing the need for costly visual annotations. The novel deep-proxy method demonstrates robustness to noisy language model outputs, outperforming existing attribute learning methods in zero-shot visual concept learning.

ABSTRACT

We present a machine learning system that can quantify fine art paintings with a set of visual elements and principles of art. This formal analysis is fundamental for understanding art, but developing such a system is challenging. Paintings have high visual complexities, but it is also difficult to collect enough training data with direct labels. To resolve these practical limitations, we introduce a novel mechanism, called proxy learning, which learns visual concepts in paintings though their general relation to styles. This framework does not require any visual annotation, but only uses style labels and a general relationship between visual concepts and style. In this paper, we propose a novel proxy model and reformulate four pre-existing methods in the context of proxy learning. Through quantitative and qualitative comparison, we evaluate these methods and compare their effectiveness in quantifying the artistic visual concepts, where the general relationship is estimated by language models; GloVe or BERT. The language modeling is a practical and scalable solution requiring no labeling, but it is inevitably imperfect. We demonstrate how the new proxy model is robust to the imperfection, while the other models are sensitively affected by it.

Motivation & Objective

• Address the scarcity of direct visual annotations for artistic elements like color, line, and texture in fine art.
• Develop a machine learning framework that learns visual concepts without visual annotations, relying only on style labels and language models.
• Evaluate the robustness of different proxy learning methods under imperfect language model supervision.
• Demonstrate that deep-proxy, a novel method combining deep CNNs with language model-provided style-concept relations, is more resilient to language model errors than existing approaches.

Proposed method

• Introduce 'proxy learning'—a framework to learn visual concepts via their general, linear relations to styles, using only style labels and language models.
• Reformulate four existing attribute learning methods (sparse coding, logistic regression, PCA, ESZSL) within the proxy learning paradigm.
• Propose a new method, 'deep-proxy', which uses a deep CNN trained solely as a style classifier but leverages language model-estimated style-concept relations to extract visual concept embeddings.
• Use pre-trained language models (GloVe and BERT) to generate embedding matrices $ G $ that encode the relationship between styles and visual concepts.
• Train the deep-CNN to predict style, while constraining its last hidden layer to align with the language model’s concept embeddings via a linear projection.
• Optimize the model using a regularized loss that enforces consistency between predicted style and proxy concept representations.

Experimental results

Research questions

• RQ1 Can visual concepts in art be effectively learned without direct visual annotations by leveraging their general relations to artistic styles?
• RQ2 How does the performance of proxy learning methods vary under imperfect language model supervision?
• RQ3 Which proxy learning method is most robust to noisy or inaccurate language model estimates of style-concept relationships?
• RQ4 To what extent can a deep CNN learn visual semantics of artistic elements when trained only on style classification with proxy supervision?

Key findings

• The deep-proxy method achieves the highest AUC scores (0.78 on eval-TEST) for visual concept prediction using BERT embeddings, outperforming other proxy methods.
• Singular Value Decomposition (SVD) in the proxy learning framework shows superior robustness to language model imperfections, with AUC scores of 0.73 (BERT) and 0.78 (GloVe) on the eval-VAL split.
• Logistic regression (LGT) is highly sensitive to language model quality, with AUC dropping to 0.38 (BERT) but rising to 0.77 (GloVe), indicating strong dependence on embedding quality.
• The Pearson correlation between language model quality scores and AUC results is higher for LGT (r ≈ 0.65) than for SVD (r ≈ 0.55), confirming SVD’s resilience to noisy language models.
• For the word 'planar', BERT incorrectly encodes negative associations with Impressionism and Expressionism, which severely affects LGT (AUC: 0.38) but only moderately affects SVD (AUC: 0.73).
• In a 12-training, 8-test style split, SVD consistently outperforms LGT, with AUC scores of 0.71 (SVD) vs. 0.63 (LGT) for the top 15 visual concepts.