[Paper Review] EFloat: Entropy-coded Floating Point Format for Deep Learning.

Rajesh Bordawekar, Bülent Abali+1 more|arXiv (Cornell University)|Feb 4, 2021

Numerical Methods and Algorithms42 references4 citations

TL;DR

EFloat is a novel floating-point format that leverages entropy coding of frequently occurring exponent values and signs to reduce average exponent field width, freeing bits for increased mantissa precision. This achieves 4–6 additional effective precision bits over standard formats like FP32, BFloat16, and 8-bit floats, enabling higher accuracy in low-precision deep learning models with reduced memory usage.

ABSTRACT

We describe the EFloat floating-point number format with 4 to 6 additional bits of precision and a wider exponent range than the existing floating point (FP) formats of any width including FP32, BFloat16, IEEE-Half precision, DLFloat, TensorFloat, and 8-bit floats. In a large class of deep learning models we observe that FP exponent values tend to cluster around few unique values which presents entropy encoding opportunities. The EFloat format encodes frequent exponent values and signs with Huffman codes to minimize the average exponent field width. Saved bits then become available to the mantissa increasing the EFloat numeric precision on average by 4 to 6 bits compared to other FP formats of equal width. The proposed encoding concept may be beneficial to low-precision formats including 8-bit floats. Training deep learning models with low precision arithmetic is challenging. EFloat, with its increased precision may provide an opportunity for those tasks as well. We currently use the EFloat format for compressing and saving memory used in large NLP deep learning models. A potential hardware implementation for improving PCIe and memory bandwidth limitations of AI accelerators is also discussed.

Motivation & Objective

• To address the memory and bandwidth bottlenecks in large deep learning models by designing a more efficient floating-point format.
• To exploit the entropy distribution of exponent values in deep learning workloads, where certain exponents occur far more frequently than others.
• To increase effective precision in low-precision formats (e.g., 8-bit, 16-bit) without increasing bit width, using entropy coding of exponents and signs.
• To enable more accurate training and inference in low-precision deep learning by reducing quantization error through improved precision.
• To provide a practical format for compressing and storing large NLP model weights with minimal precision loss.

Proposed method

• The EFloat format uses Huffman coding to encode the most frequent exponent values and sign combinations, reducing their representation cost.
• By assigning shorter codes to high-frequency exponent and sign patterns, the average width of the exponent field is reduced below that of standard floating-point formats.
• The saved bits from the compressed exponent field are reallocated to the mantissa, increasing the effective precision by 4–6 bits on average.
• The format is designed to be compatible with existing deep learning frameworks and hardware, enabling deployment in model compression and memory savings.
• The method is applied to existing floating-point formats of equal width, such as FP32, BFloat16, and 8-bit floats, to enhance their precision without increasing bit width.
• A potential hardware implementation is proposed to improve PCIe and memory bandwidth limitations in AI accelerators by reducing data movement.

Experimental results

Research questions

• RQ1 Can entropy coding of exponent values and signs in floating-point formats reduce the average field width and free bits for increased mantissa precision?
• RQ2 To what extent can EFloat improve effective precision over standard low-precision formats like BFloat16 and 8-bit floats?
• RQ3 How does EFloat perform in real-world deep learning workloads, particularly in large NLP models, in terms of memory compression and accuracy?
• RQ4 What is the potential impact of EFloat on memory bandwidth and data movement in AI accelerators?
• RQ5 Can EFloat enable more accurate training in low-precision deep learning by reducing quantization error?

Key findings

• EFloat achieves an average of 4–6 additional effective precision bits compared to standard floating-point formats of the same width, such as FP32 and BFloat16.
• The format reduces the average exponent field width by exploiting the high frequency of certain exponent values in deep learning workloads, enabling bit savings.
• The saved bits are reallocated to the mantissa, increasing numeric precision without increasing the total bit width.
• EFloat enables more accurate model compression and storage, particularly beneficial for large NLP models with high memory demands.
• The format shows potential for hardware acceleration by reducing memory bandwidth pressure through lower data movement.
• The entropy coding approach is applicable to 8-bit floats and other low-precision formats, extending its utility to emerging low-precision AI workloads.