[Paper Review] Understanding the Fisher Vector: a multimodal part model

David Novotný, Diane Larlus+2 more|arXiv (Cornell University)|Apr 18, 2015

Advanced Image and Video Retrieval Techniques17 references5 citations

TL;DR

This paper proposes that Fisher Vector (FV) object detectors function as part-based models, where each part captures multimodal appearances through a fixed set of visual words. Despite using a small codebook, FV achieves strong performance by implicitly modeling complex appearance variations via multimodal distributions, and exhibits patch- and component-level sparsity, enabling significant computational acceleration with minimal accuracy loss.

ABSTRACT

Fisher Vectors and related orderless visual statistics have demonstrated excellent performance in object detection, sometimes superior to established approaches such as the Deformable Part Models. However, it remains unclear how these models can capture complex appearance variations using visual codebooks of limited sizes and coarse geometric information. In this work, we propose to interpret Fisher-Vector-based object detectors as part-based models. Through the use of several visualizations and experiments, we show that this is a useful insight to explain the good performance of the model. Furthermore, we reveal for the first time several interesting properties of the FV, including its ability to work well using only a small subset of input patches and visual words. Finally, we discuss the relation of the FV and DPM detectors, pointing out differences and commonalities between them.

Motivation & Objective

• To understand how Fisher Vector (FV) detectors achieve strong performance despite using small visual codebooks and coarse geometric information.
• To investigate whether FV detectors can be interpreted as part-based models, similar to Deformable Part Models (DPMs).
• To reveal and quantify the sparsity properties of FV detectors at both patch and visual word levels.
• To clarify the similarities and differences between FV and DPM in modeling object appearance variations.
• To evaluate the robustness of FV detectors to pruning of patches and Gaussian components without significant performance degradation.

Proposed method

• Re-implemented the FV detection pipeline from prior work, using SIFT features and a Gaussian Mixture Model (GMM) with hard assignments to assign patches to components.
• Computed first- and second-order statistics (Fisher vector components) per patch and component using normalized differences and squared deviations from GMM means.
• Applied group lasso regularization to the FV classifier weights to induce sparsity at the Gaussian component level, encouraging removal of uninformative components.
• Used a two-stage training process: first group lasso to identify relevant components, then fine-tuned with standard ℓ₂-regularized SVM to recover performance.
• Evaluated sparsity by removing increasing percentages of low-scoring patches and Gaussian components, measuring impact on mean Average Precision (mAP).
• Visualized FV detectors to analyze part-like structures and multimodal appearance modeling, comparing them to DPM visualizations.

Experimental results

Research questions

• RQ1 Can Fisher Vector detectors be meaningfully interpreted as part-based models, with parts capturing complex, multimodal appearances?
• RQ2 How does the FV model capture diverse object appearances using a limited visual codebook and without explicit spatial deformation?
• RQ3 To what extent can FV detectors be pruned in terms of patches and Gaussian components without significant performance loss?
• RQ4 What are the key similarities and differences between FV and DPM in terms of part modeling and sparsity?
• RQ5 How does the combination of group lasso regularization and fine-tuning preserve performance during sparsity induction in FV models?

Key findings

• The FV detector exhibits patch-level sparsity: removing up to 80% of the lowest-scoring patches has minimal impact on detection performance, as measured by mAP.
• The FV detector shows component-level sparsity: up to 50% of Gaussian components can be removed with only a minor drop in mAP, especially when combined with fine-tuning after group lasso regularization.
• The FV model captures multimodal appearances through a product of distributions per part, implicitly modeling an exponential number of appearance combinations, which explains its robustness to appearance variation.
• Despite lacking explicit spatial deformation, the FV detector achieves performance comparable to DPMs by leveraging a fixed set of parts with rich, multimodal appearance modeling.
• The combination of group lasso regularization and ℓ₂ fine-tuning enables effective sparsity induction, preserving detection accuracy while allowing proportional acceleration in inference.
• Visualizations confirm that FV detectors contain interpretable, part-like structures, supporting the interpretation of FV as a multimodal part model.