[Paper Review] Modeling Image Composition for Complex Scene Generation

Zuopeng Yang, Daqing Liu+3 more|arXiv (Cornell University)|Jun 2, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a Transformer with Focal Attention (TwFA) that disentangles object-level relationships and patch-level instance structures in complex scene generation by leveraging spatial layout priors. By restricting attention to only relevant tokens—object tokens for global relationships and patch tokens within object bounding boxes for local structures—TwFA achieves state-of-the-art FID scores of 22.15 on COCO-stuff and 17.74 on Visual Genome, while enabling effective few-shot generation with minimal data.

ABSTRACT

We present a method that achieves state-of-the-art results on challenging (few-shot) layout-to-image generation tasks by accurately modeling textures, structures and relationships contained in a complex scene. After compressing RGB images into patch tokens, we propose the Transformer with Focal Attention (TwFA) for exploring dependencies of object-to-object, object-to-patch and patch-to-patch. Compared to existing CNN-based and Transformer-based generation models that entangled modeling on pixel-level&patch-level and object-level&patch-level respectively, the proposed focal attention predicts the current patch token by only focusing on its highly-related tokens that specified by the spatial layout, thereby achieving disambiguation during training. Furthermore, the proposed TwFA largely increases the data efficiency during training, therefore we propose the first few-shot complex scene generation strategy based on the well-trained TwFA. Comprehensive experiments show the superiority of our method, which significantly increases both quantitative metrics and qualitative visual realism with respect to state-of-the-art CNN-based and transformer-based methods. Code is available at https://github.com/JohnDreamer/TwFA.

Motivation & Objective

• Address the challenge of generating photo-realistic, complex scenes with multiple objects, accurate relationships, and fine-grained textures.
• Overcome the limitations of entangled modeling in CNN-based and Transformer-based methods that simultaneously learn patch-level and object-level representations.
• Improve data efficiency in few-shot learning by disentangling object-level and patch-level composition modeling.
• Enable high-fidelity, controllable image generation from layout conditions (bounding boxes and categories) with minimal supervision.

Proposed method

• Compress input RGB images into discrete patch tokens using a pre-trained compression model.
• Introduce a Transformer with Focal Attention (TwFA) that distinguishes between object tokens and patch tokens for separate modeling.
• Apply object-level attention where each object token attends to all other object tokens to capture global spatial relationships.
• Apply patch-level attention where each patch token attends only to the object it belongs to and other patches within the same object’s bounding box.
• Enforce attention constraints using a layout-guided connectivity matrix to ensure only relevant tokens are attended, reducing noise and improving disambiguation.
• Fine-tune the pre-trained TwFA model on a few images of a novel object category to enable few-shot complex scene generation.

Experimental results

Research questions

• RQ1 Can a self-attention mechanism be restructured to disentangle object-level relationships from patch-level instance structures in image generation?
• RQ2 Does incorporating spatial layout priors into attention mechanisms improve generation quality and data efficiency?
• RQ3 Can a disentangled attention mechanism enable effective few-shot image generation for novel object categories with minimal training data?
• RQ4 How does focal attention compare to standard self-attention and grid-based attention in modeling complex scene compositions?

Key findings

• TwFA achieves a new state-of-the-art FID score of 22.15 on the COCO-stuff dataset, a 25.1% improvement over the previous SOTA of 29.56.
• On the Visual Genome dataset, TwFA reduces FID from 19.14 to 17.74, a 7.3% relative improvement.
• In few-shot L2I generation, TwFA with 50 shots achieves an FID of 31.53 and an object-specific FID (Obj-FID) of 26.73, outperforming the baseline Grid16* (FID: 37.47, Obj-FID: 32.81).
• Ablation studies confirm that object-object interaction is critical for global structure reasoning, and that object-patch and patch-patch interactions significantly improve photorealism.
• The proposed focal attention mechanism outperforms grid-based attention (Grid4, Grid16) and achieves superior visual quality, especially in preserving fine details and object integrity.
• Visualizations show that TwFA generates sharper, more structured objects (e.g., faces, zebras, penguins) compared to baselines, even with only 2–50 shots of novel object data.