A Comparative Evaluation of AI-powered Binary Code Similarity Detection Approaches

Core Concepts

Despite the promising advancements of AI-powered binary code similarity detection (BinSD) techniques, particularly those based on graph neural networks (GNNs), there remains significant room for improvement, especially in addressing the "embedding collision" problem and enhancing their performance in real-world applications like vulnerability search.

Abstract

• Bibliographic Information: Fu, L., Liu, P., Meng, W., Lu, K., Zhou, S., Zhang, X., Chen, W., & Ji, S. (2024). Understanding the AI-powered Binary Code Similarity Detection. arXiv preprint arXiv:2410.07537v1.
• Research Objective: This paper presents a systematic evaluation of state-of-the-art AI-powered BinSD approaches, aiming to understand their performance in similar function detection and downstream applications, analyze the strengths and limitations of embedding neural networks and evaluation methodologies, and explore promising future research directions.
• Methodology: The authors conducted a comprehensive comparison of 15 representative AI-powered BinSD systems on similar function detection and two downstream applications: vulnerability search and license violation detection. They evaluated the accuracy and efficiency of these systems using various metrics, including AUC, accuracy, precision, recall, F1-score, Rank-1, MAP, MRR, and NDCG.
• Key Findings:
  • GNN-based BinSD approaches demonstrate the best performance in similar function detection, but they suffer from the "embedding collision" problem.
  • The performance of BinSD approaches varies significantly across different downstream applications.
  • Existing evaluation methodologies, particularly the reliance on ROC and AUC, are insufficient to accurately represent the real-world performance of BinSD tools.
  • Cross-architecture BinSD remains a significant challenge.
• Main Conclusions: The BinSD problem is far from being solved, and existing solutions require substantial improvements. The authors suggest several promising research directions, including addressing the "embedding collision" problem, developing more robust evaluation methodologies, and exploring alternative embedding neural networks.
• Significance: This study provides valuable insights into the current state of AI-powered BinSD, highlighting its limitations and potential areas for future research. The findings have significant implications for improving the accuracy and reliability of BinSD tools, which are crucial for various security and software engineering applications.
• Limitations and Future Research: The study primarily focuses on function-level BinSD approaches and does not cover the analysis of obfuscated binaries or malware. Future research could explore the applicability of these findings to other granularities of BinSD and investigate techniques for handling obfuscated code.

Stats

The GNN-based BinSD approaches achieve top-level ranking metrics in the current literature.
Compared to mono-ISA, recall values decrease significantly in cross-ISA. For instance, the recall@5 value of Gemini-skip decreases from 62.3% to 25.42% when the evaluation setting changes from mono-seen to cross-seen.
Though BinaryAI-bert2 achieves the best AUC (99.2%) and ACC (94.9%), its precision (32.21%) is 12.3% less than Gemini-skip.

Quotes

Key Insights Distilled From

Understanding the AI-powered Binary Code Similarity Detection

by Lirong Fu, P... at arxiv.org 10-11-2024

https://arxiv.org/pdf/2410.07537.pdf

Deeper Inquiries

How can the "embedding collision" problem be effectively addressed in BinSD, potentially through techniques like embedding concatenation or graph alignment?

The "embedding collision" problem, where distinct binary functions are mapped to similar embedding vectors, poses a significant challenge in AI-powered Binary Code Similarity Detection (BinSD). Addressing this issue requires enhancing the discriminative power of the embedding models. Here's how embedding concatenation and graph alignment can help:
1. Embedding Concatenation:

Concept: This approach involves extracting multiple types of features from binary code, generating separate embeddings for each feature set, and then concatenating them into a single, richer embedding vector.
Benefits: By combining diverse feature representations, such as:

Control Flow Graph (CFG) Embeddings: Capturing the function's control flow structure.
Data Flow Graph (DFG) Embeddings: Representing data dependencies within the function.
Instruction Sequence Embeddings: Encoding the sequence of instructions.
Semantic Feature Embeddings: Incorporating information about function calls, variable types, etc.
we can create more informative embeddings that better distinguish semantically different functions, reducing collision probability.

Example: A BinSD system could use a Graph Neural Network (GNN) to generate a CFG embedding and a Convolutional Neural Network (CNN) to produce an instruction sequence embedding. These two embeddings are then concatenated to represent the function.
2. Graph Alignment:

Concept: Instead of directly comparing embedding vectors, graph alignment techniques aim to find the best possible mapping between the nodes of two CFGs (or other graph representations of the binary functions).
Benefits: This approach goes beyond simple vector similarity and considers the structural correspondence between functions. By aligning nodes that represent semantically equivalent basic blocks, even if the code is structured differently, we can achieve a more accurate similarity assessment.
Example: Algorithms like the Hungarian algorithm or more advanced graph matching networks can be employed to find the optimal alignment between two CFGs. The similarity score can then be derived from the quality of the alignment.
Additional Considerations:

Feature Engineering: Careful selection and design of features are crucial for both embedding concatenation and graph alignment. Features should be robust to code variations introduced by compilers and optimization levels.
Model Selection and Training: The choice of neural network architectures and training strategies significantly impacts the quality of embeddings. Exploring more advanced GNN variants or hybrid models could further improve discriminative power.
By combining these techniques and continuously refining feature representations and embedding models, we can mitigate the "embedding collision" problem and enhance the accuracy and reliability of AI-powered BinSD systems.