Skip to main content
QUICK REVIEW

[论文解读] FLUXSynID: A Synthetic Face Dataset with Document and Live Images

Jianzhu Guo, Dingyun Zhang|arXiv (Cornell University)|Jul 3, 2024
Human Motion and AnimationEngineering被引用 9
一句话总结

本论文提出一种非扩散、基于隐式关键点的视频驱动人像动画框架,名为 LivePortrait,并通过拼接与重定向模块来实现高质量、可控且高效的动画。

ABSTRACT

FLUXSynID: A Synthetic Face Dataset with Document and Live Images FLUXSynID is a high-resolution synthetic identity dataset containing 14,889 unique synthetic identities, each represented through a document-style image and three live capture variants. Identities are generated using the FLUX.1 [dev] diffusion model, guided by user-defined identity attributes such as gender, age, region of origin, and other various identity features. The dataset is created to support biometric research, including face recognition and morphing attack detection. File Structure Each identity has a dedicated folder (named as a 12-digit hex string, e.g., 000e23cdce23) containing the following 5 files: 000e23cdce23_f.json — metadata including sampled identity attributes, prompt, generation seed, etc. (_f = female; _m = male; _nb = non-binary) 000e23cdce23_f_doc.png — document-style frontal image 000e23cdce23_f_live_0_e_d1.jpg — live image generated with LivePortrait (_e = expression and pose) 000e23cdce23_f_live_0_a_d1.jpg — live image via Arc2Face (_a = arc2face) 000e23cdce23_f_live_0_p_d1.jpg — live image via PuLID (_p = pulid) All document and LivePortrait/PuLID images are 1024×1024. Arc2Face images are 512×512 due to original model constraints. Attribute Sampling and Prompting The attributes/ directory contains all information about how identity attributes were sampled: A set of .txt files (e.g., ages.txt, eye_shape.txt, body_type.txt) — each lists the possible values for one attribute class, along with their respective sampling probabilities. file_probabilities.json — defines the inclusion probability for each attribute class (i.e., how likely a class such as "eye shape" is to be included in a given prompt). attribute_clashes.json — specifies rules for resolving semantically conflicting attributes. Each clash defines a primary attribute (to be kept) and secondary attributes (to be discarded when the clash occurs). Prompts are generated automatically using Qwen2.5 large language model, based on selected attributes, and used to condition FLUX.1 [dev] during image generation. Live Image Generation Each synthetic identity has three live image-style variants: LivePortrait: expression/pose changes via keypoint-based retargeting Arc2Face: natural variation using identity embeddings (no prompt required) PuLID: identity-aware generation using prompt, embedding, and edge-conditioning with a customized FLUX.1 [dev] diffusion model These approaches provide both controlled and naturalistic identity-consistent variation. Filtering and Quality Control Included are 9 supplementary text files listing filtered subsets of identities. For instance, file similarity_filtering_adaface_thr_0.333987832069397_fmr_0.0001.txt contains identities retained after filtering out overly similar faces using AdaFace FRS under the specified threshold and false match rate (FMR). Usage and Licensing This dataset is licensed under the Creative Commons Attribution Non Commercial 4.0 International (CC BY-NC 4.0) license.You are free to use, share, and adapt the dataset for non-commercial purposes, provided that appropriate credit is given. The images in this dataset were generated using the FLUX.1 [dev] model by Black Forest Labs, which is made available under their Non-Commercial License. While this dataset does not include or distribute the model or its weights, the images were produced using that model. Users are responsible for ensuring that their use of the images complies with the FLUX.1 [dev] license, including any restrictions it imposes. Acknowledgments The FLUXSynID dataset was developed under the EINSTEIN project. The EINSTEIN project is funded by the European Union (EU) under G.A. no. 101121280 and UKRI Funding Service under IFS reference 10093453. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect the views of the EU/Executive Agency or UKRI. Neither the EU nor the granting authority nor UKRI can be held responsible for them.

研究动机与目标

  • 以单一源图像为基础,利用驱动信号(视频、运动或生成)实现高效、可控的人像动画的研究目标。
  • 在保持高保真度和高效性的同时,提升对风格化人像与跨身份场景的泛化能力。
  • 开发用于提升可控性且计算开销可忽略的拼接与重定向等机制。

提出的方法

  • 在 Face Vid2vid 的基础上,采用具有增强的非扩散式隐式关键点框架。
  • 采用混合图像–视频训练策略与大规模数据集(约 ~69M 帧)以提升泛化能力。
  • 将规范关键点学习、头部姿态和表情形变统一为一个网络,配合基于 SPADE 的生成器。
  • 引入可拓展的运动变换,带有显式尺度因子以在驱动过程中稳定纹理。
  • 通过在眼部和嘴部关键点上应用 Wing 损失,对以地标为引导的隐式关键点进行优化。
  • 实现级联感知损失与 GAN 损失以及面部身份损失,以保持身份并提升纹理质量。
  • 引入拼接模块和两个重定向模块(眼睛与嘴唇)以在几乎无开销的情况下提升可控性。
  • 拼接模块在粘贴回原始图像空间时对不同身份与区域的运动进行对齐。

实验结果

研究问题

  • RQ1如何利用隐式关键点实现高质量、可控且高效的人像动画?
  • RQ2混合图像–视频训练是否能提升对风格化或跨身份人像的泛化能力?
  • RQ3拼接与重定向模块是否能在不带来显著计算成本的情况下提升可控性?

主要发现

  • Ours 模型在自重演和跨重演任务中,达到或优于若干非扩散与扩散基线的定量指标。
  • 拼接使肩部对齐保持一致,并在粘贴回原始图像空间时减少空间错位。
  • 眼睛与嘴唇的重定向模块提供可控、保持身份的调整,提升跨身份驱动下的表情保真度。
  • 在 RTX 4090 上推理速度达到 12.8 ms,展示了相对于扩散方法的显著效率优势。
  • 使用来自 Voxceleb、MEAD、RAVDESS、AAHQ,以及风格化 LightStage 派生数据的混合图像–视频训练策略,获得对风格化人像的更好泛化。

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。