Systematic Evaluation of Graph Benchmarking Practices Reveals Significant Flaws

Core Concepts

Widespread inconsistencies and flaws in benchmarking practices for graph processing systems lead to misleading and non-reproducible results.

Abstract

The authors conducted a 12-year literary review of graph processing benchmarking practices and found significant issues:

Lack of standardization in benchmarking practices, with a wide variety of datasets, benchmarks, and metrics used across the literature.
Inconsistent use of datasets, with different papers reporting different vertex and edge counts for the same datasets.
Overuse of synthetic graph generators that produce graphs with unrealistic characteristics, distorting performance results.
Significant impact of dataset properties, such as vertex ordering and presence of zero-degree vertices, on benchmark performance, which is often ignored.

The authors then conducted a quantitative study to demonstrate the severity of these issues. They showed that:

Vertex ordering can cause up to 38% performance differences in PageRank on popular graph processing systems.
The presence of zero-degree vertices can lead to a 10x performance boost for benchmarks like BFS and Triangle Counting.
Different graph processing systems report different numbers of triangles for the same directed graph dataset, due to a lack of standardization in triangle counting definitions.

The authors conclude by proposing a set of best practices for benchmarking graph processing systems, including:

Developing a standardized set of benchmarks and datasets
Using the Smooth Kronecker graph generator for synthetic datasets
Reporting detailed preprocessing steps and metrics
Specifying vertex ordering and triangle counting definitions
Selecting appropriate datasets for each benchmark

Stats

Changing the vertex ID assignment of the Twitter2010 dataset can cause a performance difference of up to 38% for the PageRank benchmark on several popular graph processing systems.
The presence of isolated vertices in the citPatents dataset can cause a 10x performance boost for benchmarks such as BFS and Connected Components.

Quotes

"Evaluations frequently ignore datasets' statistical idiosyncrasies, which significantly affect system performance."
"Scalability studies often use datasets that fit easily in memory on a modest desktop."
"Currently, the community has no consistent and principled manner with which to compare systems and provide guidance to developers who wish to select the system most suited to their application."

Key Insights Distilled From

SoK

by Puneet Mehro... at arxiv.org 04-02-2024

https://arxiv.org/pdf/2404.00766.pdf

Deeper Inquiries

How can the graph processing community incentivize the adoption of the proposed best practices for benchmarking?

To incentivize the adoption of best practices for benchmarking in the graph processing community, several strategies can be implemented. Firstly, organizing workshops, tutorials, and seminars focused on benchmarking practices can raise awareness and educate researchers on the importance of standardized benchmarks and datasets. Providing resources such as guidelines, templates, and tools for benchmarking can make it easier for researchers to adhere to best practices. Additionally, creating incentives such as awards or recognition for research that follows best practices can motivate researchers to adopt these standards. Collaboration with industry partners to promote the use of standardized benchmarks in real-world applications can also drive adoption within the community.

What are the potential challenges in developing a standardized set of benchmarks and datasets for graph processing systems?

Developing a standardized set of benchmarks and datasets for graph processing systems can face several challenges. One major challenge is the diversity of graph data and the complexity of real-world graphs, making it difficult to create benchmarks that are representative of all possible scenarios. Ensuring the quality and relevance of the datasets used in benchmarks can be challenging, as datasets need to capture the characteristics of real-world graphs accurately. Another challenge is the dynamic nature of graph data, requiring benchmarks to adapt to evolving graphs over time. Additionally, gaining consensus and buy-in from the research community to adopt and adhere to standardized benchmarks can be a challenge, as researchers may have different preferences and priorities when evaluating graph processing systems.

How can the graph processing community better incorporate the evolving nature of real-world graphs into their benchmarking practices?

To better incorporate the evolving nature of real-world graphs into benchmarking practices, the graph processing community can implement several strategies. One approach is to develop benchmarking frameworks that support dynamic graph data, allowing researchers to evaluate systems on graphs that change over time. Incorporating streaming data and continuous updates into benchmarking scenarios can provide a more realistic evaluation of system performance. Additionally, promoting research on dynamic graph algorithms and datasets can help researchers understand and address the challenges posed by evolving graphs. Collaboration with industry partners who work with real-time graph data can also provide insights and datasets that reflect the dynamic nature of graph processing applications.