[논문 리뷰] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
저자들은 Pick-a-Pic을 공개된 실사용자 프롬프트와 텍스트-이미지 생성에 대한 이미지 선호도의 대규모 공개 데이터 세트로 발표하고, PickScore를 훈련시켰다. PickScore는 CLIP 기반 채점 함수로 인간 선호도를 초인간 수준의 정확도로 예측하고 기존 메트릭보다 모델 평가에 뛰어나다.
The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.
연구 동기 및 목표
- Create a large, open dataset of real user prompts and preferences in text-to-image generation.
- Train a scoring function (PickScore) to predict user preferences from prompt–image pairs.
- Demonstrate PickScore’s superiority for model evaluation over traditional metrics like FID.
- Show how ranking with PickScore can improve image quality and model selection.
- Encourage adoption of Pick-a-Pic and PickScore for future T2I research and evaluation.
제안 방법
- Develop a web app that lets users generate images from prompts and indicate preferences between two images per round.
- Collect and preprocess real user interactions to form the Pick-a-Pic dataset with prompts, image pairs, and preferences.
- Fine-tune a CLIP-H backbone with a reward-model–style objective to maximize the likelihood of preferred images (KL-divergence minimization).
- Train PickScore by combining a CLIP-based text/image encoder with a reward-model objective and a temperature parameter, using a dataset-weighted loss to handle prompt frequency.
- Evaluate PickScore against baselines (CLIP-H, aesthetics predictor, random, and human experts) using adapted accuracy with ties and tie-threshold analysis.
- Compare model evaluation signals (PickScore vs FID) and show correlations with human judgments on MS-COCO captions.
실험 결과
연구 질문
- RQ1Can Pick-a-Pic provide a realistic, open distribution of prompts and preferences reflecting real user needs?
- RQ2Does PickScore reliably predict human preferences better than existing automatic scoring functions?
- RQ3Is PickScore a better metric for evaluating and ranking text-to-image models than FID or other baselines?
- RQ4Can ranking via PickScore meaningfully improve the quality of generated images compared to other scoring methods?
주요 결과
- Pick-a-Pic delivers a large open dataset with over 500k examples (and later versions over 1M) of prompts, image pairs, and user preferences.
- PickScore achieves 70.5% accuracy in predicting human preferences on the validation/test sets, exceeding human annotators (68.0%) and baselines.
- PickScore shows a strong correlation with real user judgments (0.917 vs. FID’s -0.900 correlation on MS-COCO prompts).
- PickScore outperforms CLIP-H, an aesthetics predictor, and ImageReward/HPS baselines on the Pick-a-Pic evaluation.
- Using prompts from Pick-a-Pic yields more human-aligned model evaluation than MS-COCO captions.
- In image ranking experiments, PickScore-selected images are preferred by humans more often than selections by other scoring methods.]
- table_headers:["Model","Accuracy"]
- table_rows:[["Random","56.8"],["Human Expert","68.0"],["Aesthetics [14]","56.8"],["CLIP-H [7]","60.8"],["ImageReward [18]","61.1"],["HPS [17]","66.7"],["PickScore (Ours)","70.5"]]}?} {
이 논문 읽기
누빈트 뷰어에서 원문을 열고, 읽으면서 궁금한 곳을 바로 물어보세요.
누빈트에서 이 논문 읽기이 리뷰는 AI가 논문 원문을 바탕으로 만들었습니다. 인용하기 전에 원문을 확인해 주세요.