Efficient Distributed Algorithms for Scalable String Sorting

Core Concepts

The authors present practical and efficient algorithms for distributed-memory string sorting that scale to large numbers of processors. The algorithms achieve speedups of up to 5 over the current state-of-the-art distributed string sorting algorithms.

Abstract

The paper presents new algorithms for distributed-memory string sorting that aim to address the scalability limitations of existing approaches. The key contributions are: Multi-Level Merge Sort (MS): Recursively splits the processors into groups, with each group solving an independent string sorting problem. Achieves a trade-off between latency and communication volume by allowing the data to be communicated k times. The maximum number of strings and characters per processor is bounded, enabling good scalability. Multi-Level Prefix Doubling Merge Sort (PDMS): Approximates the distinguishing prefixes of strings using a multi-level distributed single-shot Bloom filter. Only exchanges the approximate distinguishing prefixes, reducing the communication volume. Combines the prefix doubling approach with the multi-level merge sort algorithm. The paper also presents an improved hypercube quicksort algorithm for strings and a multi-level distributed single-shot Bloom filter as side results. The experimental evaluation shows that the multi-level algorithms outperform the single-level state-of-the-art algorithms, achieving speedups of up to 5 on a wide range of inputs on up to 49,152 cores.

Stats

The total number of strings is n. The total number of characters is N. The length of the longest string is ˆl. The length of the shortest string is ˇl. The length of the longest distinguishing prefix is ˆd. The sum of the lengths of all distinguishing prefixes is D.

Quotes

"String sorting is an important part of tasks such as building index data structures. Unfortunately, current string sorting algorithms do not scale to massively parallel distributed-memory machines since they either have latency (at least) proportional to the number of processors p or communicate the data a large number of times (at least logarithmic)." "We present practical and efficient algorithms for distributed-memory string sorting that scale to large p. Similar to state-of-the-art sorters for atomic objects, the algorithms have latency of about p^(1/k) when allowing the data to be communicated k times."

Key Insights Distilled From

Scalable Distributed String Sorting

by Florian Kurp... at arxiv.org 04-26-2024

https://arxiv.org/pdf/2404.16517.pdf

Deeper Inquiries

How can the proposed algorithms be extended to support space-efficient string sorting, where the strings are highly compressible and cannot be fully materialized in memory?

The proposed algorithms can be extended to support space-efficient string sorting by incorporating techniques that allow for sorting strings of equal length without materializing all strings at the same time. One approach to achieve this is by implementing a suffix sorting algorithm based on sorting strings of equal length. This method involves sorting the suffixes of a text with a combined length of approximately n^2/2 using only n characters. To handle highly compressible strings where many strings overlap, the algorithms can utilize techniques such as prefix doubling merge sort with grid-wise Bloom filters for duplicate detection. By approximating the distinguishing prefixes of strings and efficiently detecting duplicates, the algorithms can reduce the amount of data that needs to be stored in memory during the sorting process. Additionally, incorporating LCP compression and optimizing the communication volume can further enhance the space efficiency of the sorting algorithms.

How can the proposed algorithms be extended to support space-efficient string sorting, where the strings are highly compressible and cannot be fully materialized in memory?

The proposed algorithms can be extended to support space-efficient string sorting by incorporating techniques that allow for sorting strings of equal length without materializing all strings at the same time. One approach to achieve this is by implementing a suffix sorting algorithm based on sorting strings of equal length. This method involves sorting the suffixes of a text with a combined length of approximately n^2/2 using only n characters. To handle highly compressible strings where many strings overlap, the algorithms can utilize techniques such as prefix doubling merge sort with grid-wise Bloom filters for duplicate detection. By approximating the distinguishing prefixes of strings and efficiently detecting duplicates, the algorithms can reduce the amount of data that needs to be stored in memory during the sorting process. Additionally, incorporating LCP compression and optimizing the communication volume can further enhance the space efficiency of the sorting algorithms.

How can the proposed algorithms be extended to support space-efficient string sorting, where the strings are highly compressible and cannot be fully materialized in memory?

The proposed algorithms can be extended to support space-efficient string sorting by incorporating techniques that allow for sorting strings of equal length without materializing all strings at the same time. One approach to achieve this is by implementing a suffix sorting algorithm based on sorting strings of equal length. This method involves sorting the suffixes of a text with a combined length of approximately n^2/2 using only n characters. To handle highly compressible strings where many strings overlap, the algorithms can utilize techniques such as prefix doubling merge sort with grid-wise Bloom filters for duplicate detection. By approximating the distinguishing prefixes of strings and efficiently detecting duplicates, the algorithms can reduce the amount of data that needs to be stored in memory during the sorting process. Additionally, incorporating LCP compression and optimizing the communication volume can further enhance the space efficiency of the sorting algorithms.