[论文解读] AI Matrix - Synthetic Benchmarks for DNN

Wei Wei, Lingjie Xu+3 more|arXiv (Cornell University)|Nov 27, 2018

Advanced Neural Network Applications参考文献 4被引用 5

一句话总结

本文提出 AI Matrix，一种合成基准测试框架，通过分析真实应用的特征，生成最小化且具代表性的深度神经网络（DNN）工作负载，实现自适应、高效且无版权问题的硬件性能评估。该框架通过计算模式的统计建模，显著减少基准测试数量与运行时间，同时准确反映多样化 DNN 工作负载的特性。

ABSTRACT

Deep neural network (DNN) architectures, such as convolutional neural networks (CNN), involve heavy computation and require hardware, such as CPU, GPU, and AI accelerators, to provide the massive computing power. With the many varieties of AI hardware prevailing on the market, it is often hard to decide which one is the best to use. Thus, benchmarking AI hardware effectively becomes important and is of great help to select and optimize AI hardware. Unfortunately, there are few AI benchmarks available in both academia and industry. Examples are BenchNN[1], DeepBench[2], and Dawn Bench[3], which are usually a collection of typical real DNN applications. While these benchmarks provide performance comparison across different AI hardware, they suffer from a number of drawbacks. First, they cannot adapt to the emerging changes of DNN algorithms and are fixed once selected. Second, they contain tens to hundreds of applications and take very long time to finish running. Third, they are mainly selected from open sources, which are restricted by copyright and are not representable to proprietary applications. In this work, a synthetic benchmarks framework is firstly proposed to address the above drawbacks of AI benchmarks. Instead of pre-selecting a set of open-sourced benchmarks and running all of them, the synthetic approach generates only a one or few benchmarks that best represent a broad range of applications using profiled workload characteristics data of these applications. Thus, it can adapt to emerging changes of new DNN algorithms by re-profiling new applications and updating itself, greatly reduce benchmark count and running time, and strongly represent DNN applications of interests. The generated benchmarks are called AI Matrix, serving as a performance benchmarks matching the statistical workload characteristics of a combination of applications of interests.

研究动机与目标

• 解决现有 DNN 基准测试的局限性，包括固定不变、规模庞大且常受版权限制的问题。
• 通过生成反映真实世界 DNN 工作负载特征的合成基准测试，实现高效且具代表性的硬件性能评估。
• 通过重新分析新应用的特征，支持对新兴 DNN 架构的动态适应。
• 在保持多样化 DNN 工作负载高度代表性的同时，减少所需基准测试的数量。
• 为现有开源基准测试套件提供可扩展、可重用且法律安全的替代方案。

提出的方法

• 分析真实 DNN 应用，提取统计工作负载特征，如层类型、张量形状和计算强度。
• 利用采集的数据，合成一组最小化且具代表性的基准测试——AI Matrix，使其匹配目标工作负载的统计分布。
• 基于采集的统计数据，生成具有可控架构与计算特性的合成 DNN 模型。
• 确保合成基准测试保留关键性能特征，如 FLOPs、内存访问模式和层间连接性。
• 通过重新分析新 DNN 模型并相应调整合成生成过程，动态更新基准测试集。
• 通过在多个硬件平台上对比 AI Matrix 与真实 DNN 应用的性能表现，验证 AI Matrix 的代表性。

实验结果

研究问题

• RQ1 合成基准测试框架能否有效代表广泛真实 DNN 应用的性能特征？
• RQ2 AI Matrix 在保持硬件比较准确性的前提下，能在多大程度上减少基准测试数量与执行时间？
• RQ3 AI Matrix 在无需手动重新配置的情况下，对新 DNN 架构的适应能力如何？
• RQ4 AI Matrix 是否可作为现有开源 DNN 基准测试的合法安全且可扩展的替代方案？
• RQ5 在计算与内存行为方面，AI Matrix 对真实世界 DNN 工作负载的代表性如何？

主要发现

• AI Matrix 显著减少了所需基准测试的数量，同时保持了对真实 DNN 工作负载的高度代表性。
• 与包含数百个模型的传统基准测试套件相比，该框架通过最小化执行时间，显著缩短了基准测试周期。
• 通过重新分析，AI Matrix 能够适应新 DNN 架构，使其紧跟深度学习技术的发展趋势。
• AI Matrix 生成的合成基准测试在多种硬件平台上，其性能趋势与真实 DNN 应用高度一致。
• 该方法通过不依赖现有模型权重或代码，避免了版权问题，适用于专有及商业用途。
• 评估结果表明，AI Matrix 提供了与完整基准测试套件相当可靠的硬件排名与性能洞察。