[Paper Review] BLT: Bidirectional Layout Transformer for Controllable Layout Generation

Xiang Rong Kong, Lu Jiang+5 more|arXiv (Cornell University)|Dec 9, 2021

Handwritten Text Recognition Techniques4 citations

TL;DR

BLT introduces a bidirectional layout transformer that enables controllable, non-autoregressive layout generation by attending to attributes in both directions during training and iteratively refining low-confidence attributes at inference. It achieves up to 10x faster inference than autoregressive baselines while maintaining state-of-the-art performance on six diverse layout generation benchmarks, with a novel hierarchical sampling policy critical for high-quality results.

ABSTRACT

Creating visual layouts is a critical step in graphic design. Automatic generation of such layouts is essential for scalable and diverse visual designs. To advance conditional layout generation, we introduce BLT, a bidirectional layout transformer. BLT differs from previous work on transformers in adopting non-autoregressive transformers. In training, BLT learns to predict the masked attributes by attending to surrounding attributes in two directions. During inference, BLT first generates a draft layout from the input and then iteratively refines it into a high-quality layout by masking out low-confident attributes. The masks generated in both training and inference are controlled by a new hierarchical sampling policy. We verify the proposed model on six benchmarks of diverse design tasks. Experimental results demonstrate two benefits compared to the state-of-the-art layout transformer models. First, our model empowers layout transformers to fulfill controllable layout generation. Second, it achieves up to 10x speedup in generating a layout at inference time than the layout transformer baseline. Code is released at https://shawnkx.github.io/blt.

Motivation & Objective

• To address the limitation of autoregressive layout transformers in conditional layout generation due to immutable attribute dependency chains.
• To enable flexible, user-controlled layout generation where any attribute can be modified based on input constraints.
• To improve inference speed while maintaining or improving layout quality compared to autoregressive transformers.
• To develop a hierarchical sampling policy that enhances non-autoregressive layout generation by capturing structural correlations between layout attributes.
• To validate the model across diverse design applications, including GUIs, magazines, ads, and home decoration.

Proposed method

• BLT employs a non-autoregressive transformer architecture that predicts masked attributes by attending to surrounding attributes in both forward and backward directions during training.
• During inference, BLT generates a draft layout and iteratively refines low-confidence attributes using a hierarchical sampling policy to guide mask selection.
• The hierarchical sampling policy organizes attributes into groups (category, size, position) and controls mask generation based on a predefined order, improving structural consistency.
• The model uses a bidirectional self-attention mechanism that allows context from both earlier and later attributes to inform predictions, breaking the rigid autoregressive dependency chain.
• Training involves masking specific attribute groups and optimizing for reconstruction, while inference uses iterative refinement with confidence-based masking.
• The method is evaluated using standard layout metrics including IoU, overlap, alignment, FID, and similarity, across six diverse datasets.

Experimental results

Research questions

• RQ1 Can a non-autoregressive transformer model achieve high-quality layout generation while enabling controllable, conditional layout editing?
• RQ2 Does a bidirectional attention mechanism improve layout generation quality compared to unidirectional autoregressive models?
• RQ3 How effective is the proposed hierarchical sampling policy in guiding mask generation for non-autoregressive layout refinement?
• RQ4 Can the model achieve significant speedups over autoregressive baselines without sacrificing layout quality?
• RQ5 Does the model generalize across diverse design domains such as GUIs, magazines, and advertisements?

Key findings

• BLT achieves up to 10x faster inference than autoregressive transformer baselines, with speed advantage increasing on dense layouts (e.g., 10x speedup for 20 objects).
• The model outperforms state-of-the-art autoregressive and VAE-based models in conditional layout generation, particularly in handling user-defined constraints on layout attributes.
• The hierarchical sampling policy significantly improves performance over standard non-autoregressive baselines, reducing IoU by 0.07 and overlap by 0.10 on the RICO dataset.
• On the PubLayNet dataset, BLT reduces FID from 217 (non-autoregressive baseline) to 134, demonstrating improved layout quality.
• The model maintains competitive performance on unconditional layout generation, achieving FID of 70 on RICO and 134 on PubLayNet, comparable to state-of-the-art models.
• Ablation studies confirm that the hierarchical sampling order (e.g., category → size → position) is critical, with suboptimal orders leading to higher IoU and alignment scores.