[Paper Review] A very preliminary analysis of DALL-E 2

Gary Marcus, Ernest Davis+1 more|arXiv (Cornell University)|Apr 25, 2022

Handwritten Text Recognition TechniquesComputer Science88 citations

TL;DR

The paper reports a very preliminary evaluation of DALL-E 2 using fourteen challenging prompts, finding that at least one image met the request for 5 prompts, but no prompt yielded all ten images that satisfied the request.

ABSTRACT

The DALL-E 2 system generates original synthetic images corresponding to an input text as caption. We report here on the outcome of fourteen tests of this system designed to assess its common sense, reasoning and ability to understand complex texts. All of our prompts were intentionally much more challenging than the typical ones that have been showcased in recent weeks. Nevertheless, for 5 out of the 14 prompts, at least one of the ten images fully satisfied our requests. On the other hand, on no prompt did all of the ten images satisfy our requests.

Motivation & Objective

• Assess DALL-E 2's ability to handle prompts requiring common sense and reasoning.
• Test understanding of complex textual prompts beyond typical showcase prompts.
• Provide a preliminary, quantitative sense of DALL-E 2's performance on challenging tasks.

Proposed method

• Design fourteen prompts significantly more challenging than typical demonstrations.
• Generate ten images per prompt to assess consistency and satisfaction.
• Evaluate whether generated images meet the specific demands of each prompt.

Experimental results

Research questions

• RQ1 Can DALL-E 2 reliably satisfy prompts that test common sense and reasoning?
• RQ2 To what extent does DALL-E 2 understand and execute complex textual requirements across multiple images per prompt?

Key findings

• Out of fourteen prompts, at least one image satisfied the request for five prompts.
• For none of the prompts did all ten images satisfy the request.
• The prompts used were deliberately more challenging than those showcased publicly.
• The study provides a very early, limited view of DALL-E 2's capabilities.