[論文レビュー] UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes

А. И. Колесников, André Susano Pinto+4 more|arXiv (Cornell University)|May 20, 2022

Advanced Neural Network Applications被引用数 23

ひとこと要約

tldr: UViM は、学習された離散コードで誘導される基盤のフィードフォワードモデルと、その guiding code を生成する自己回帰言語モデルを組み合わせた2段階の統一ビジョンモデルであり、タスク固有のアーキテクチャを使わずにパノプティック分割、深度予測、カラー化で競争力のある結果を実現します。

ABSTRACT

We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks. In contrast to previous models, UViM has the same functional form for all tasks; it requires no task-specific modifications which require extensive human expertise. The approach involves two components: (I) a base model (feed-forward) which is trained to directly predict raw vision outputs, guided by a learned discrete code and (II) a language model (autoregressive) that is trained to generate the guiding code. These components complement each other: the language model is well-suited to modeling structured interdependent data, while the base model is efficient at dealing with high-dimensional outputs. We demonstrate the effectiveness of UViM on three diverse and challenging vision tasks: panoptic segmentation, depth prediction and image colorization, where we achieve competitive and near state-of-the-art results. Our experimental results suggest that UViM is a promising candidate for a unified modeling approach in computer vision.

研究の動機と目的

• Motivate a unified approach to diverse vision tasks with high-dimensional structured outputs.
• Eliminate task-specific architectural tweaks by introducing a learned guiding code framework.
• Demonstrate that a shared base model plus an autoregressive code model can tackle segmentation, depth, and colorization.
• Show that end-to-end two-stage training yields competitive, near-state-of-the-art results.

提案手法

• Introduce a two-stage training procedure: Stage I trains a base model guided by a restricted oracle that outputs a short discrete guiding code z from the ground truth y.
• Stage II trains an autoregressive language model to predict the guiding code z from the input x, enabling f(x, LM(x)) to perform the task.
• Use a discrete bottleneck inspired by VQ-VAE to learn z and apply a dictionary-learning update (LBFGS-like) to prevent underutilized codebook entries.
• Parameterize f and the restricted oracle Omega with ViT; LM is an encoder-decoder Transformer with ViT encoder and Transformer decoder.
• Train jointly end-to-end in Stage I, then train LM to mimic Omega’s output in Stage II; at test time, compute z = LM(x) and predict y = f(x, z).
• Discuss code dropout during Stage II to improve robustness by randomly zeroing parts of z during training.

実験結果

リサーチクエスチョン

• RQ1 Can a single, uniform modeling framework produce competitive results across diverse vision tasks with high-dimensional structured outputs?
• RQ2 Does introducing a learned guiding code and an autoregressive LM enable efficient modeling of complex output dependencies without task-specific modifications?
• RQ3 To what extent does the two-stage training (oracle-guided base model plus LM-generated guiding code) generalize across panoptic segmentation, depth estimation, and colorization?
• RQ4 What are the trade-offs in guiding-code length and dictionary size for Stage I, and how do code dropout and autoregressive modeling affect final performance?

主な発見

• UViM achieves competitive results across three diverse tasks (panoptic segmentation, depth prediction, and colorization) without task-specific architectures.
• Stage I with a restricted oracle plus a VQ-VAE-like discrete bottleneck enables the base model to solve high-dimensional structured outputs when aided by the guiding code.
• Stage II trains an autoregressive LM to predict the guiding code from the image, enabling a unified model to handle different tasks with a single pipeline.
• Ablations show autoregressive modeling of the guiding code is crucial; removing it degrades performance significantly.
• Using pre-trained weights and code dropout improves final performance and robustness; from-scratch training remains competitive but slower.
• Code length and dictionary size affect performance; longer sequences and larger dictionaries help Stage I, with a sweet spot for the final model.
• Compared to task-specific baselines, UViM is near state-of-the-art on the evaluated tasks and demonstrates strong transferability and generality.