Core Concepts
Innovative runtime optimization techniques using Adaptive Metaprogramming significantly improve the performance of recursive queries.
Abstract
This paper introduces Adaptive Metaprogramming, a novel technique that shifts recursive query optimization from compile-time to runtime using principled metaprogramming. The study evaluates the optimization potential of Adaptive Metaprogramming through Carac, a custom Datalog engine, showing significant improvements in unoptimized recursive query execution time. The research highlights the importance of dynamic optimization and re-optimization for recursive query processing.
I. Introduction:
Traditional relational algebra-based query optimization struggles with recursive queries due to unpredictable relation cardinalities.
Datalog is popular for high-performance recursive queries in various applications.
II. Background:
Datalog is rule-based declarative query language used in logic programming.
Multi-Stage Programming enables compilation in multiple phases at compile-time and runtime.
III. Adaptive Metaprogramming Architecture:
Staged approach allows continuous re-optimization of recursive queries at runtime.
Concrete instances of relations are dynamically plugged into join reordering algorithm for optimal performance.
IV. Runtime Join-Order Optimization:
Reordering atoms within Datalog rules significantly impacts program performance.
Carac uses Adaptive Metaprogramming to dynamically regenerate query plans based on runtime information.
V. Carac: System Design:
Carac integrates specialized Datalog execution engine with general-purpose programming language using Multi-Stage Programming.
VI. Evaluation:
A. Carac Query Benchmarks:
Macrobenchmarks include static analysis and data flow analysis queries on real-world datasets like httpd dataset.
Microbenchmarks cover shorter-running programs like Ackermann function and Fibonacci program to evaluate online optimizations' impact on faster programs.
B. Online Optimization Speedup:
JIT optimizations significantly improve unoptimized programs' performance, showcasing the effectiveness of Adaptive Metaprogramming techniques.
Different compilation targets offer trade-offs between expressibility, maintainability, and performance.
Stats
Runtime information informs query optimization - "An optimal query plan will not stay optimal over the execution of a single program."
Adaptive Metaprogramming improves unoptimized recursive query execution time by three orders of magnitude and hand-optimized queries by 6x.
Quotes
"Optimization decisions can be made or revised as soon as new information becomes available."
"Staging allows gradual specialization for making optimization decisions based on concrete information."