[Paper Review] Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph

Dacheng Yin, Xuanchi Ren+4 more|arXiv (Cornell University)|Feb 24, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

Retriever proposes a modal-agnostic, unsupervised framework for learning content-style disentangled representations by modeling them as a token-level bipartite graph, using cross-attention for style retrieval, vector quantization with man-induced constraints for interpretable content tokens, and a novel link attention module for reconstruction. It achieves state-of-the-art zero-shot voice conversion and competitive part discovery in images, demonstrating strong disentanglement and generative potential.

ABSTRACT

This paper addresses the unsupervised learning of content-style decomposed representation. We first give a definition of style and then model the content-style representation as a token-level bipartite graph. An unsupervised framework, named Retriever, is proposed to learn such representations. First, a cross-attention module is employed to retrieve permutation invariant (P.I.) information, defined as style, from the input data. Second, a vector quantization (VQ) module is used, together with man-induced constraints, to produce interpretable content tokens. Last, an innovative link attention module serves as the decoder to reconstruct data from the decomposed content and style, with the help of the linking keys. Being modal-agnostic, the proposed Retriever is evaluated in both speech and image domains. The state-of-the-art zero-shot voice conversion performance confirms the disentangling ability of our framework. Top performance is also achieved in the part discovery task for images, verifying the interpretability of our representation. In addition, the vivid part-based style transfer quality demonstrates the potential of Retriever to support various fascinating generative tasks. Project page at https://ydcustc.github.io/retriever-demo/.

Motivation & Objective

• To establish a unified, modal-agnostic definition of content and style in structured data based on permutation invariance.
• To develop an unsupervised framework that decomposes content and style without modality-specific or dataset-specific definitions.
• To enable fine-grained, interpretable manipulation of content and style through a learnable bipartite graph structure.
• To achieve high-fidelity data reconstruction and downstream performance in both speech and vision tasks.

Proposed method

• Define style as permutation-invariant (P.I.) information at the token level, and content as the remaining order-sensitive information.
• Use a cross-attention module with a shared prototype to retrieve P.I. style tokens from input data.
• Apply vector quantization (VQ) with man-induced constraints to produce discrete, interpretable content tokens.
• Introduce a link attention module that uses linking keys to reconstruct data from content and style via content-query-driven style retrieval.
• Train the model end-to-end using a combination of reconstruction loss, VQ diversity loss, and structural constraint loss.
• Ensure modal-agnosticity by designing components that operate on token-level representations without modality-specific inductive biases.

Experimental results

Research questions

• RQ1 Can a unified, permutation-invariant definition of style be applied across modalities like speech and images?
• RQ2 How can content and style be disentangled in an unsupervised manner without relying on paired data or modality-specific supervision?
• RQ3 Can a bipartite graph structure at the token level enable fine-grained, interpretable control over content and style for downstream tasks?
• RQ4 Does the proposed link attention mechanism effectively reconstruct data from separated content and style representations?
• RQ5 Can the framework generalize to real-world, diverse datasets such as LibriSpeech beyond curated benchmarks?

Key findings

• Retriever achieves state-of-the-art zero-shot voice conversion performance with 99.6% similarity accuracy on the CMU Arctic dataset when using 10 target utterances.
• On the LibriSpeech dataset, the model reaches 98.4% speaker verification accuracy and 3.13 MOSNet score with 60 style tokens, demonstrating robustness in complex, real-world settings.
• The ablation study confirms that both structural constraint and VQ diversity loss are essential, as their absence reduces speech quality (MOSNet drops to 2.94).
• The model is robust to hyperparameter variation: SV accuracy fluctuates by less than ±0.45% and MOSNet by less than ±0.02 across a wide range of loss weight settings.
• Part-level style transfer in images produces vivid, interpretable results, confirming the interpretability of the learned content and style representations.
• The framework achieves competitive performance in image part discovery, validating the interpretability of the content tokens derived via man-induced constraints.