[论文解读] Fast exact summation using small and large superaccumulators

Radford M. Neal|arXiv (Cornell University)|May 21, 2015

Numerical Methods and Algorithms参考文献 7被引用 11

一句话总结

本文提出一种利用混合小规模和大规模超累加器（固定精度整数）的快速精确求和方法，用于计算双精度浮点数的精确和，并正确舍入到最近的浮点数表示值。该方法实现接近内存带宽限制的性能，现代64位处理器上耗时不足简单求和的两倍，从而在不损失可复现性的情况下，实现高效串行与并行精确求和。

ABSTRACT

I present two new methods for exactly summing a set of floating-point numbers, and then correctly rounding to the nearest floating-point number. Higher accuracy than simple summation (rounding after each addition) is important in many applications, such as finding the sample mean of data. Exact summation also guarantees identical results with parallel and serial implementations, since the exact sum is independent of order. The new methods use variations on the concept of a "superaccumulator" - a large fixed-point number that can exactly represent the sum of any reasonable number of floating-point values. One method uses a "small" superaccumulator with sixty-seven 64-bit chunks, each with 32-bit overlap with the next chunk, allowing carry propagation to be done infrequently. The small superaccumulator is used alone when summing a small number of terms. For big summations, a "large" superaccumulator is used as well. It consists of 4096 64-bit chunks, one for every possible combination of exponent bits and sign bit, plus counts of when each chunk needs to be transferred to the small superaccumulator. To add a term to the large superaccumulator, only a single chunk and its associated count need to be updated, which takes very few instructions if carefully implemented. On modern 64-bit processors, exactly summing a large array using this combination of large and small superaccumulators takes less than twice the time of simple, inexact, ordered summation, with a serial implementation. A parallel implementation using a small number of processor cores can be expected to perform exact summation of large arrays at a speed that reaches the limit imposed by memory bandwidth. Some common methods that attempt to improve accuracy without being exact may therefore be pointless, at least for large summations, since they are slower than computing the sum exactly.

研究动机与目标

• 开发一种快速、精确的浮点数求和方法，确保结果正确舍入到最近的可表示值。
• 克服现有精确求和技术在处理大数组时的性能瓶颈。
• 在串行与并行实现中均实现可复现的、与顺序无关的结果。
• 降低小规模问题中精确求和的固定计算开销。
• 支持高精度统计计算，例如在R语言中精确计算样本均值。

提出的方法

• 使用一个‘小’超累加器，包含67个重叠的64位块，每一块与前一块有32位重叠，以减少频繁进位传播的频率。
• 使用一个‘大’超累加器，包含4096个块——每个符号和指数位组合对应一个块——每个块记录其被更新的次数。
• 通过仅修改一个块及其关联计数来更新大超累加器，从而最小化指令数量。
• 仅在需要时才将大超累加器中的非零块转移到小超累加器，以减少开销。
• 结合使用两种超累加器：小超累加器用于小规模求和，大超累加器用于大规模求和，并在并行计算中高效合并部分结果。
• 使用整数算术以定点格式表示精确和，避免累加过程中出现浮点数舍入误差。

实验结果

研究问题

• RQ1 在现代64位处理器上，能否以接近内存带宽限制的速度实现精确求和？
• RQ2 混合超累加器设计能否降低小数组中精确求和的固定成本？
• RQ3 使用按指数位跟踪的大规模超累加器是否能实现精确求和的高效并行化？
• RQ4 在大规模问题中，精确求和是否能在准确性和性能上均优于不精确但更快的方法？
• RQ5 精确求和的性能是否受内存带宽限制而非计算能力限制，尤其是在并行环境中？

主要发现

• 混合小规模与大规模超累加器的方法在现代64位处理器上实现精确求和的耗时不足简单求和的两倍。
• 对于大数组，该方法达到内存带宽极限，表明性能不再受CPU限制，而是受数据移动限制。
• 在4个核心下，该方法可在Intel Xeon E3-1230 v2上实现峰值内存带宽利用率，表明在大规模数据集上仅需少数核心即可达到最佳性能。
• 由于固定初始化和扫描开销，小超累加器方法在极小规模求和中比简单求和慢10–20倍。
• 大超累加器支持高效并行化：将数组分割并在并行中分别求和，结果与串行执行完全一致，确保了可复现性。
• 该方法表明，对于大规模应用，非精确但更快的求和技术可能并非必要，因为精确方法既能保证准确性又能实现高效性能。