[Paper Review] Automatic Image Cropping for Visual Aesthetic Enhancement Using Deep Neural Networks and Cascaded Regression

Guanjun Guo, Hanzi Wang+3 more|arXiv (Cornell University)|Dec 25, 2017

Visual Attention and Saliency Detection31 references6 citations

TL;DR

This paper proposes a cascaded cropping regression (CCR) method that leverages deep CNN features and gradient-boosted random-ferns regressors to automatically crop images for enhanced visual aesthetics. By using a two-step learning strategy—first training a CNN on large-scale aesthetic datasets and then regressing cropping boxes from CNN features—the method achieves superior performance over state-of-the-art approaches, with 4.3–6.3% failure rates and 1.7s per-image inference time.

ABSTRACT

Despite recent progress, computational visual aesthetic is still challenging. Image cropping, which refers to the removal of unwanted scene areas, is an important step to improve the aesthetic quality of an image. However, it is challenging to evaluate whether cropping leads to aesthetically pleasing results because the assessment is typically subjective. In this paper, we propose a novel cascaded cropping regression (CCR) method to perform image cropping by learning the knowledge from professional photographers. The proposed CCR method improves the convergence speed of the cascaded method, which directly uses random-ferns regressors. In addition, a two-step learning strategy is proposed and used in the CCR method to address the problem of lacking labelled cropping data. Specifically, a deep convolutional neural network (CNN) classifier is first trained on large-scale visual aesthetic datasets. The deep CNN model is then designed to extract features from several image cropping datasets, upon which the cropping bounding boxes are predicted by the proposed CCR method. Experimental results on public image cropping datasets demonstrate that the proposed method significantly outperforms several state-of-the-art image cropping methods.

Motivation & Objective

• Address the challenge of automatic image cropping for visual aesthetic enhancement, where aesthetic quality is subjective and difficult to quantify.
• Overcome the scarcity of labeled cropping data by leveraging large-scale weakly labeled aesthetic datasets to pre-train a CNN before cropping regression.
• Improve convergence speed and accuracy of cascaded regression by using gradient-boosted random-ferns as primitive regressors.
• Develop a two-step learning pipeline that first extracts discriminative features via a pre-trained CNN and then regresses optimal cropping regions.
• Achieve high-quality cropping results that align closely with professional photographers' annotations, even with limited bounding box supervision.

Proposed method

• Train a deep convolutional neural network (CNN) classifier on a large-scale aesthetic dataset (AVA) to learn discriminative image representations.
• Extract CNN features from image crops using the pre-trained CNN, using these features as input for the cropping regression task.
• Implement a cascaded regression framework where each stage uses a group of weak random-ferns regressors, refined via gradient boosting to improve convergence.
• Use a two-stage learning strategy: pre-train the CNN on image-level aesthetic labels, then fine-tune the regression head using bounding-box annotations from professional photographers.
• Apply the trained CCR model to predict optimal cropping regions by iteratively refining bounding box predictions through multiple regression stages.
• Optimize the regression process by leveraging the high-level semantic features from the CNN to guide the localization of aesthetically pleasing regions.

Experimental results

Research questions

• RQ1 Can a two-step learning strategy effectively bridge the gap between weakly labeled aesthetic data and limited bounding-box cropping annotations?
• RQ2 How does cascaded regression with gradient-boosted random-ferns improve convergence and accuracy compared to standard cascaded regression with random-ferns?
• RQ3 To what extent can CNN-extracted features from a pre-trained aesthetic classifier improve the performance of automatic image cropping?
• RQ4 How does the proposed method compare to state-of-the-art methods in terms of IoU and BDE metrics on public cropping datasets?
• RQ5 What types of images are most challenging for the proposed method, and what failure patterns emerge?

Key findings

• The proposed CCR method significantly outperforms four state-of-the-art methods in both IoU and BDE metrics across three public image cropping datasets.
• The method achieves a failure rate of only 4.3–6.3% on the three test datasets, indicating high reliability in predicting professional-quality crop regions.
• The average inference time per image is 1.7 seconds, which is substantially faster than the 11 seconds reported for a competing improved change-based method.
• The attention-based method performed worst due to its focus on salient objects while neglecting overall aesthetic composition.
• The aesthetic-based method underperformed because it prioritized high-quality local regions over holistic composition, unlike the CCR method which learns from professional cropping patterns.
• Failure cases primarily occur when objects are near image boundaries (risk of partial cropping) or when backgrounds are uniform and textureless, suggesting limitations in edge and structure modeling.