Constrained Horn Clause Solver Competition 2023: Results and Insights

Core Concepts

The CHC-COMP 2023 competition evaluated state-of-the-art solvers for Constrained Horn Clauses, a formalism widely used in program verification. The competition featured seven solvers and six tracks, each focusing on a class of CHCs with different background theories and constraints.

Abstract

The CHC-COMP 2023 was the sixth edition of the annual competition for Constrained Horn Clause (CHC) solvers. The competition featured seven solvers, six of which were competing and one was entered as hors concours. The competition was organized into six tracks, each dealing with a class of CHCs with different background theories and constraints:

LIA-lin: Linear Integer Arithmetic - linear clauses
LIA-nonlin: Linear Integer Arithmetic - nonlinear clauses
LIA-lin-Arrays: Linear Integer Arithmetic & Arrays - linear clauses
LIA-nonlin-Arrays: Linear Integer Arithmetic & Arrays - nonlinear clauses
LIA-nonlin-Arrays-nonrecADT: Linear Integer Arithmetic & Arrays & non-recursive Algebraic Data Types - nonlinear clauses
ADT-LIA-nonlin: Algebraic Data Types & Linear Integer Arithmetic - nonlinear clauses

The competition was run on the StarExec platform, with each solver given a time limit of 1800 seconds for the competition runs. The results were evaluated based on the number of satisfiable and unsatisfiable benchmarks solved by each solver in each track. In case of a tie, the total CPU time was used to determine the ranking.
The winners of the competition were:

LIA-lin: Golem
LIA-nonlin, LIA-lin-Arrays, LIA-nonlin-Arrays, LIA-nonlin-Arrays-nonrecADT, ADT-LIA-nonlin: Eldarica
The report also discusses some issues encountered during the competition runs and how they were resolved in collaboration with the participants. Finally, it outlines some open issues and proposals for future editions of the competition.

Stats

None

Quotes

None

Key Insights Distilled From

CHC-COMP 2023: Competition Report

by Emanuele De ... at arxiv.org 04-24-2024

https://arxiv.org/pdf/2404.14923.pdf

Deeper Inquiries

How could the CHC-COMP competition be further improved to better evaluate the capabilities and limitations of the participating solvers

To further improve the CHC-COMP competition and better evaluate the capabilities and limitations of participating solvers, several enhancements can be considered:

Include Parallel Tracks: Introducing parallel tracks where solvers can run with a limit on wall-clock time instead of CPU time can provide insights into the scalability and efficiency of solvers in handling larger problem instances.

Validation of Results: Implementing a common format for generating and presenting witnesses can enhance result validation. This would allow for a more thorough analysis of solver performance and correctness.

Status of Benchmarks: Requiring benchmarks to explicitly declare the expected result of the satisfiability problem can help in verifying the correctness of solver outputs and benchmark suitability.

Increased Benchmark Diversity: Encouraging the submission of a broader range of benchmarks, including those with complex constraints and background theories, can provide a more comprehensive evaluation of solver capabilities.

Feedback Mechanism: Establishing a feedback mechanism where participants receive detailed feedback on their solver performance, including areas for improvement, can foster continuous development and innovation in CHC solving techniques.

What are the key challenges in developing efficient CHC solvers that can handle a wide range of background theories and constraints

Developing efficient CHC solvers that can handle a wide range of background theories and constraints poses several key challenges:

Handling Nonlinear Constraints: Dealing with nonlinear constraints in CHCs requires sophisticated algorithms and techniques to ensure scalability and accuracy in solving complex problems.

Integration of Background Theories: Incorporating diverse background theories such as linear integer arithmetic, arrays, and algebraic data types into CHC solvers necessitates robust integration mechanisms and efficient constraint solving strategies.

Scalability: Ensuring scalability to handle large-scale CHC problems with numerous constraints and variables is crucial for practical applications in program verification and beyond.

Optimization and Performance: Optimizing solver performance to achieve fast and accurate results while minimizing resource consumption is a constant challenge in CHC solver development.

Handling Recursion: Addressing recursion in CHCs effectively requires specialized techniques to analyze and reason about recursive structures within the constraints.

What are the potential applications of CHC solving beyond program verification, and how could the CHC-COMP competition be adapted to encourage research in these areas

The potential applications of CHC solving extend beyond program verification to areas such as cybersecurity, software synthesis, and AI. To encourage research in these areas, the CHC-COMP competition could be adapted in the following ways:

Diversification of Tracks: Introduce new tracks focusing on cybersecurity challenges, software synthesis problems, and AI applications to attract researchers from these domains.

Incorporate Real-World Scenarios: Include benchmarks inspired by real-world scenarios in sectors like cybersecurity and AI to showcase the practical relevance of CHC solving techniques.

Collaboration with Industry: Foster collaborations with industry partners to provide industry-specific problem sets and challenges for the competition, promoting the development of solutions with direct applicability.

Special Awards or Recognition: Introduce special awards or recognition categories for innovative applications of CHC solving beyond traditional program verification, encouraging participants to explore new avenues for research and development.