[Paper Review] A Bijective String Sorting Transform

Hendrian, Diptarama, Köppl, Dominik+2 more|arXiv (Cornell University)|Jan 15, 2012

Algorithms and Data Compression10 references36 citations

TL;DR

This paper introduces a bijective string sorting transform (S-transform) that, unlike the standard Burrows-Wheeler Transform (BWT), is invertible for all input strings without requiring a sentinel character or rotation index. The S-transform groups identical characters together through a novel sorting mechanism based on cyclic suffixes and permutations, achieving better compression ratios than BWT on the Calgary corpus, with gains up to 11.37% on the PIC file and an average improvement of 3.05% across all files, while maintaining linear time and space complexity for both transform and inverse operations.

ABSTRACT

Given a string of characters, the Burrows-Wheeler Transform rearranges the characters in it so as to produce another string of the same length which is more amenable to compression techniques such as move to front, run-length encoding, and entropy encoders. We present a variant of the transform which gives rise to similar or better compression value, but, unlike the original, the transform we present is bijective, in that the inverse transformation exists for all strings. Our experiments indicate that using our variant of the transform gives rise to better compression ratio than the original Burrows-Wheeler transform. We also show that both the transform and its inverse can be computed in linear time and consuming linear storage.

Motivation & Objective

• To develop a bijective string transform that is invertible for all strings, unlike the standard BWT which requires additional metadata like a sentinel or rotation index.
• To improve compression efficiency by grouping identical characters more effectively than BWT while preserving the structural advantages of the BWT.
• To design a transform that is both computationally efficient and suitable for applications requiring data confidentiality, such as encryption of compressed data.
• To formalize and prove correctness of a novel transform based on sorting rotations of substrings derived from successive minimal suffixes.
• To demonstrate that the inverse transform can be computed in linear time using a generalized cycle-traversal algorithm.

Proposed method

• The S-transform decomposes the input string into substrings by iteratively selecting and removing the lexicographically smallest suffix.
• All rotations of these substrings are generated and sorted lexicographically as a single list.
• The transform output is formed by taking the last character of each sorted rotation, resulting in a string that groups identical characters together.
• The inverse transform uses a permutation ϑ derived from the transform's structure, where ϑ maps positions in the transformed string to the previous character in the original string's cyclic order.
• A generalized cycle-traversal algorithm, MultiThread, is used to reconstruct the original string from the transformed string and permutation ϑ, processing cycles in ϑ from last to first character.
• The algorithm ensures linear time and space complexity by tracking visited positions with a temporary array and marking them as processed.

Experimental results

Research questions

• RQ1 Can a bijective string sorting transform be constructed that groups identical characters together like BWT but without requiring a sentinel character or rotation index?
• RQ2 Does such a transform achieve better or comparable compression performance compared to BWT on standard corpora?
• RQ3 Can the inverse of this transform be computed efficiently, in linear time and space, without relying on additional metadata?
• RQ4 Is the transform reversible through a deterministic, algorithmically efficient process based on permutation cycles?
• RQ5 Can the same core algorithm used in BWT inversion be adapted to work with this new transform using a generalized cycle-traversal mechanism?

Key findings

• The S-transform achieves better compression than BWT on all files in the Calgary corpus except GEO, with an average improvement of 3.05% across all files.
• On the PIC file, the S-transform achieves a compression gain of 11.37%, the highest among all files in the corpus.
• The transform outperforms BWT even in cases where the gain is minimal, such as in PROGL, where a 200-byte saving is observed despite a relative gain of only 1.02%.
• The inverse transform is correctly computed using the MultiThread algorithm, which processes cycles in the permutation ϑ to reconstruct the original string from last to first character.
• Both the S-transform and its inverse are computable in O(n) time and O(n) space, matching the efficiency of the standard BWT.
• The S-transform is bijective: every output string uniquely maps back to one input string, eliminating the need for sentinel characters or rotation indices.