[Paper Review] Muse: Text-To-Image Generation via Masked Generative Transformers

Hui‐Wen Chang, Han Zhang+10 more|arXiv (Cornell University)|Jan 2, 2023

Generative Adversarial Networks and Image Synthesis119 citations

TL;DR

Muse introduces a text-to-image Transformer that uses masked token prediction in a discrete latent space, conditioned on frozen LLM embeddings, achieving state-of-the-art FID/CLIP while enabling fast, parallel decoding and zero-shot editing.

ABSTRACT

We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io

Motivation & Objective

• Advance text-to-image synthesis by leveraging masked modeling in discrete token space.
• Incorporate pre-trained language model embeddings to improve semantic fidelity and spatial reasoning.
• Improve inference efficiency with parallel decoding over discrete tokens compared to diffusion/autoregressive baselines.
• Enable zero-shot image editing (inpainting, outpainting, mask-free editing) without fine-tuning.
• Evaluate on CC3M and COCO to demonstrate state-of-the-art quality and alignment.

Proposed method

• Encode images to discrete tokens using dual VQGAN tokenizers (256×256 with f=16; 512×512 with f=8).
• Condition the image decoder on frozen T5-XXL text embeddings to provide rich language conditioning.
• Use a base masked Transformer to predict masked image tokens via cross-attention to text embeddings and self-attention among image tokens.
• Train with variable masking rates sampled from a cosine schedule to encourage robust token prediction and enable flexible sampling.
• Follow with a super-resolution Transformer to translate low-res tokens to high-res tokens, conditioning on text embeddings.
• Apply classifier-free guidance during sampling to improve text-image alignment and allow negative prompts.
• Perform iterative parallel decoding to predict multiple tokens per step, enabling faster inference than autoregressive or diffusion models.

Figure 1 : Muse text-to-image generation ( $512\times 512$ resolution). Under each generated image, the corresponding caption is shown, exhibiting a variety of styles, captions and understanding. Each image was generated in $1.3$ s on a TPUv4 chip.

Experimental results

Research questions

• RQ1 Can a masked, discrete-token image model conditioned on pre-trained LLM embeddings achieve state-of-the-art image fidelity and text alignment?
• RQ2 How does the combination of base and super-resolution token transformers perform for 256×256 and 512×512 outputs?
• RQ3 What inference efficiency gains are achievable with parallel decoding in a discrete-token framework compared to diffusion/autoregressive baselines?
• RQ4 To what extent can zero-shot editing (inpainting, outpainting, mask-free editing) be achieved without fine-tuning?
• RQ5 How does Muse perform on CC3M and COCO in terms of FID and CLIP while maintaining fast sampling?

Key findings

• Muse achieves state-of-the-art FID on CC3M (6.06 with 632M base + 268M super-res tokens).
• Muse-3B attains COCO zero-shot FID of 7.88 with CLIP 0.32.
• Muse outperforms comparable models on human alignment prompts, with prompt-image alignment favored ~2.7× over Stable Diffusion in user studies.
• Inference is significantly faster than diffusion or autoregressive models (e.g., 0.5–1.3s per 256×256–512×512 image on TPUv4) due to discrete tokens and parallel decoding.
• Zero-shot image editing is enabled directly (inpainting, outpainting, and mask-free editing) without fine-tuning or inversion, via conditional token resampling.
• Qualitative results show strong understanding of cardinality, composition, style, and text rendering, though challenges remain for long multi-word phrases and high cardinalities.

Figure 2 : Examples of zero-shot text-guided image editing using Muse. We show examples of a number of editing applications using the Muse text-to-image generative model, on real input images, without fine-tuning. All edited images are generated at $512\times 512$ resolution.