[논문 리뷰] A Learned Representation For Artistic Style

Vincent Dumoulin, Jonathon Shlens외 1명|arXiv (Cornell University)|2016. 10. 24.

Aesthetic Perception and Analysis인용 수 749

한 줄 요약

본 논문은 conditional instance normalization을 도입하여 단일 다스타일 피드포워드 스타일 트랜스퍼 네트워크를 학습시키고, 하나의 임베딩으로 다수의 그림 스타일을 포착하며 임의의 스타일 혼합을 가능하게 하고 소수의 매개변수로 새로운 스타일을 빠르게 도입할 수 있게 한다.

ABSTRACT

The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.

연구 동기 및 목표

• Motivate learning a parsimonious representation of artistic styles beyond per-style networks.
• Show that a single network can model multiple styles via an embedding of style parameters.
• Demonstrate that new styles can be added efficiently by fine-tuning limited parameters.
• Illustrate that the embedding allows arbitrary style composition and interpolation between styles.

제안 방법

• Adopt a style transfer network architecture and train with a style loss and content loss as in neural style transfer.
• Introduce conditional instance normalization where gamma and beta are style-dependent and learned as matrices with one row per style.
• Share almost all network weights across styles while only learning style-specific affine parameters.
• Demonstrate that a new style can be added by updating a small subset of parameters while keeping others fixed.
• Show that convex combinations of style parameters yield interpolated pastiches between styles.

실험 결과

연구 질문

• RQ1 Can a single network with shared weights model multiple artistic styles effectively?
• RQ2 How does conditional instance normalization enable multi-style representation with few style-specific parameters?
• RQ3 Is it possible to add new styles efficiently by fine-tuning only style-conditional parameters?
• RQ4 Can styles be arbitrarily combined through the learned style embedding?
• RQ5 Does a multi-style network perform comparably to individually trained single-style networks?

주요 결과

• A single network trained on 10 Monet styles captures diverse color palettes and textures with 99.8% shared parameters across styles (0.2% per-style).
• A multi-style network achieves style transfer quality comparable to independently trained single-style models.
• New styles can be integrated by fine-tuning gamma and beta while keeping weights fixed, with faster convergence than training from scratch.
• The learned style embedding supports arbitrary convex combinations of styles to create novel pastiches.
• The approach scales to 32 diverse styles and remains efficient in training and memory usage.