[Paper Review] ImagineNet: Restyling Apps Using Neural Style Transfer

Michael Fischer, Richard R. Yang+1 more|arXiv (Cornell University)|Jan 14, 2020

Generative Adversarial Networks and Image Synthesis29 references4 citations

TL;DR

ImagineNet introduces a novel neural style transfer model that preserves GUI legibility by adding a cross-layer uncentered cross-covariance loss term to the style transfer objective, enabling high-fidelity restyling of apps with artistic textures while maintaining clear object boundaries and readable text. The method significantly outperforms prior techniques in user preference studies.

ABSTRACT

This paper presents ImagineNet, a tool that uses a novel neural style transfer model to enable end-users and app developers to restyle GUIs using an image of their choice. Former neural style transfer techniques are inadequate for this application because they produce GUIs that are illegible and hence nonfunctional. We propose a neural solution by adding a new loss term to the original formulation, which minimizes the squared error in the uncentered cross-covariance of features from different levels in a CNN between the style and output images. ImagineNet retains the details of GUIs, while transferring the colors and textures of the art. We presented GUIs restyled with ImagineNet as well as other style transfer techniques to 50 evaluators and all preferred those of ImagineNet. We show how ImagineNet can be used to restyle (1) the graphical assets of an app, (2) an app with user-supplied content, and (3) an app with dynamically generated GUIs.

Motivation & Objective

• To enable end-users and developers to restyle GUIs using any chosen artwork while preserving legibility and interface functionality.
• To address the critical limitation of existing neural style transfer methods, which distort GUI elements and render text illegible.
• To model style more comprehensively by incorporating structural coherence across CNN layers, beyond just texture and color.
• To evaluate the effectiveness of the approach through user studies and real-world app restyling scenarios.
• To demonstrate the feasibility of applying artistic styles to dynamic, user-generated, and third-party GUIs without breaking usability.

Proposed method

• Introduces a new loss term based on the uncentered cross-covariance of features across different layers in a pre-trained VGG19 network to model structural style.
• Modifies the standard neural style transfer objective by minimizing the squared error between the cross-covariance of the style image and the generated output image.
• Preserves content details by emphasizing structural consistency across feature hierarchies, ensuring object boundaries and text remain legible.
• Uses a multi-scale optimization framework where the model is trained with a batch size of 1 for up to 10 hours to achieve high-quality results.
• Supports real-time inference after training, with transformation times of 3.02 ms for 512×288 and 104.02 ms for 1920×1080 images.
• Allows users to cancel training early if results are unsatisfactory, with meaningful outputs visible within one minute.

Experimental results

Research questions

• RQ1 Can a neural style transfer model preserve GUI legibility and object boundaries while transferring artistic styles?
• RQ2 Does incorporating cross-layer feature correlations improve the coherence and visual quality of stylized GUIs?
• RQ3 How do users perceive the visual quality and usability of stylized GUIs generated by ImagineNet compared to existing methods?
• RQ4 Which types of artistic styles are most suitable for GUI restyling, and why?
• RQ5 Can the method be generalized across different app types, including those with static assets, user content, and dynamically generated UIs?

Key findings

• In a user study with 50 evaluators, all preferred the GUIs restyled by ImagineNet over those produced by prior style transfer methods.
• The average user rating for 96 different styles ranged from 1.6 to 4.5, with 82 out of 96 styles rated above 3, indicating strong consensus on style quality.
• Abstract Expressionism, High Renaissance, Modernismo, and Pop Art styles transferred well due to strong edges and vibrant colors.
• Action painting, Lettrism, Kinetic Art, and Purism styles performed poorly due to lack of structural clarity and distinct borders.
• The method achieved a transformation speed of 0.026 μs per pixel, with 512×288 GUIs processed in 3.02 ms and 1920×1080 in 104.02 ms on average.
• The structure loss term was essential: without it, style transfer results were incoherent and text was illegible, even with enhanced content loss.