[论文解读] MILDNet: A Lightweight Single Scaled Deep Ranking Architecture

Anirudha Vishvakarma|arXiv (Cornell University)|Mar 3, 2019

Advanced Image and Video Retrieval Techniques参考文献 26被引用 5

一句话总结

MILDNet 提出了一种轻量级、单尺度的深度排序架构，通过融合多个中间卷积神经网络层的特征，实现了当前最优的视觉相似性性能，相比多尺度CNN，模型参数量减少3倍、训练速度提升3倍、推理延迟降低3倍。该模型通过跳跃连接和消融验证的特征融合机制，在仅使用一个紧凑模型的情况下，实现了与Street-to-Shop数据集上SOTA方法相当的准确率。

ABSTRACT

Multi-scale deep CNN architecture [1, 2, 3] successfully captures both fine and coarse level image descriptors for visual similarity task, but they come up with expensive memory overhead and latency. In this paper, we propose a competing novel CNN architecture, called MILDNet, which merits by being vastly compact (about 3 times). Inspired by the fact that successive CNN layers represent the image with increasing levels of abstraction, we compressed our deep ranking model to a single CNN by coupling activations from multiple intermediate layers along with the last layer. Trained on the famous Street2shop dataset [4], we demonstrate that our approach performs as good as the current state-of-the-art models with only one third of the parameters, model size, training time and significant reduction in inference time. The significance of intermediate layers on image retrieval task has also been shown to be performing on popular datasets Holidays, Oxford, Paris [5]. So even though our experiments are done on ecommerce domain, it is applicable to other domains as well. We further did an ablation study to validate our hypothesis by checking the impact on adding each intermediate layer. With this we also present two more useful variants of MILDNet, a mobile model (12 times smaller) for on-edge devices and a compactly featured model (512-d feature embeddings) for systems with less RAMs and to reduce the ranking cost. Further we present an intuitive way to automatically create a tailored in-house triplet training dataset, which is very hard to create manually. This solution too can also be deployed as an all-inclusive visual similarity solution. Finally, we present our entire production level architecture which currently powers visual similarity at Fynd.

研究动机与目标

• 解决视觉相似性任务中多尺度CNN带来的高内存占用与高延迟问题。
• 开发一种紧凑的深度学习架构，高效捕捉图像的细粒度与粗粒度特征。
• 通过模型变体实现在内存受限设备上的部署能力。
• 自动化生成内部三元组训练数据，用于视觉相似性学习。
• 构建可扩展、可投入生产的电商视觉相似性处理流水线。

提出的方法

• MILDNet 融合单个CNN网络中多个中间层（包括最后一层）的特征，以捕捉分层抽象表征。
• 模型利用跳跃连接整合不同深度的激活值，增强特征丰富度，同时不增加网络深度。
• 训练过程中使用查询-正样本-负样本三元组，应用三元组损失函数以实现相似性学习。
• 通过消融实验验证各中间层对检索性能的贡献。
• 提出两种变体：基于MobileNet的轻量化版本，适用于边缘设备部署；以及512维特征版本，适用于内存受限系统。
• 设计自动化流水线，生成针对特定领域优化的内部三元组训练数据集，用于视觉相似性训练。

实验结果

研究问题

• RQ1 是否可通过单尺度CNN结合多层级特征融合，在视觉相似性任务中达到与多尺度CNN相当的性能？
• RQ2 引入中间层特征对检索准确率与模型效率有何影响？
• RQ3 在不牺牲准确率的前提下，模型尺寸与推理延迟最多可降低多少？
• RQ4 自动化方法能否生成高质量、领域特定的三元组训练数据用于视觉相似性学习？
• RQ5 所提出的架构在真实电商生产流水线中的实际效果如何？

主要发现

• MILDNet 在Street-to-Shop数据集上实现了SOTA性能，参数量和模型尺寸仅为当前多尺度模型的1/3。
• 与现有SOTA方法相比，模型训练时间与推理延迟均降低约3倍。
• 按顺序逐步加入中间层特征可提升检索性能，其中早期与中期特征带来的增益最为显著。
• 512维嵌入变体仅导致准确率下降2.1%，但显著提升了索引速度并降低了内存占用。
• 基于MobileNet的MILDNet变体将模型尺寸缩小12倍（至20 MB），准确率下降5.4%，实现了边缘设备的可行部署。
• 自动化三元组数据生成流水线实现了可扩展的内部训练数据创建，显著降低了人工标注成本。