[論文レビュー] Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

Shin-Ying Yeh, Yu-Guan Hsieh+4 more|arXiv (Cornell University)|Sep 26, 2023

Generative Adversarial Networks and Image Synthesis被引用数 10

ひとこと要約

この論文は LyCORIS を紹介します。Stable Diffusion の微調整のための複数のランクベース手法と、プロンプト・概念・ハイパーパラメータを横断してこれらの手法を比較する包括的な評価フレームワークを提供します。

ABSTRACT

Text-to-image generative models have garnered immense attention for their ability to produce high-fidelity images from text prompts. Among these, Stable Diffusion distinguishes itself as a leading open-source model in this fast-growing field. However, the intricacies of fine-tuning these models pose multiple challenges from new methodology integration to systematic evaluation. Addressing these issues, this paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion) [https://github.com/KohakuBlueleaf/LyCORIS], an open-source library that offers a wide selection of fine-tuning methodologies for Stable Diffusion. Furthermore, we present a thorough framework for the systematic assessment of varied fine-tuning techniques. This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning, including hyperparameter adjustments and the evaluation with different prompt types across various concept categories. Through this comprehensive approach, our work provides essential insights into the nuanced effects of fine-tuning parameters, bridging the gap between state-of-the-art research and practical application.

研究の動機と目的

• Present LyCORIS, an open-source library offering a spectrum of fine-tuning methods for Stable Diffusion.
• Propose a comprehensive evaluation framework using diverse prompts and metrics to assess fine-tuning performance.
• Investigate how hyperparameters and training choices affect concept fidelity, text-image alignment, and base-model preservation.
• Provide practical insights into when different LyCORIS methods excel for various customization tasks.

提案手法

• Introduce LyCORIS, a library implementing LoRA-based and alternative rank-adaptation methods for Stable Diffusion Fine-tuning.
• Describe core algorithms: LoRA (LoCon), LoHa, and LoKr, including their mathematical formulations (e.g., h′ = W0h + b + γ∆Wh with ∆W decomposed via B and A, and variants with Hadamard and Kronecker products).
• Define a concept descriptor with a trigger word and optional class word used in captions and prompts.
• Outline a modular evaluation framework with prompt types (train, generalization, concept-agnostic) and metrics (fidelity, controllability, diversity, base-model preservation, image quality).
• Conduct extensive experiments across 26 configurations, 3 seeds, and 3 checkpoints per config, evaluating with multiple prompt types and diverse concepts.
• Apply SHAP analysis and CatBoost to interpret how configuration choices influence evaluation metrics.]
• research_questions

実験結果

リサーチクエスチョン

• RQ1 How do different LyCORIS fine-tuning algorithms (LoRA, LoHa, LoKr) compare in terms of fidelity, controllability, diversity, and base-model preservation?
• RQ2 How do hyperparameters (learning rate, trained layers, dimension, alpha, factor) and data choices impact performance across diverse concept categories?
• RQ3 What guidance can be derived on when to use each algorithm for simple versus complex, multi-concept fine-tuning tasks?
• RQ4 How should prompts be structured to properly evaluate learned concepts beyond training captions?

主な発見

• LoHa generally excels for simple, multi-concept fine-tuning tasks, while LoKr with full dimension benefits complex, single-concept scenarios.
• Increasing model capacity (dimension) or reducing the factor can shift performance similarly to higher learning rates or longer training, with nuanced effects across metrics.
• Fine-tuning all layers (including convolutional) tends to increase image similarity but may degrade other aspects like text-image alignment and base-model preservation when hyperparameters are not balanced.
• Restrictive fine-tuning to attention layers lowers image similarity and can hinder learning target concepts, highlighting the need to tune beyond attention modules.
• A comprehensive evaluation framework reveals that there is no single superior method; performance is task- and prompt-dependent, emphasizing careful metric design and multiple prompts for robust assessment.
• The study advocates for broader, systematic evaluation in the model customization landscape beyond existing single-metric assessments.