Core Concepts
ML query processing in distributed systems benefits from Compass, reducing job latency and improving resource utilization.
Abstract
Compass introduces a novel framework for ML query processing in distributed systems. It focuses on reducing job latency by efficiently managing GPU memory and task placement. The system aims to place tasks where data dependencies are satisfied, collocate tasks from the same job, and manage GPU memory effectively. By unifying these functions, Compass shows a significant reduction in completion times compared to other state-of-the-art schedulers while using the same or fewer resources. The system is designed to handle edge clusters physically close to end-users, targeting rapid response times for interactive applications utilizing machine intelligence.
Stats
Comparison with other state of the art schedulers shows a significant reduction in completion times.
In one case, just half the servers were needed for processing the same workload.
ML model parameters can be hundreds of megabytes in size.
GPU memory is treated as a cache with the cache hit rate being an important metric.