Compass: A Decentralized Scheduler for Latency-Sensitive ML Workflows

Core Concepts

ML query processing in distributed systems benefits from Compass, reducing job latency and improving resource utilization.

Abstract

Compass introduces a novel framework for ML query processing in distributed systems. It focuses on reducing job latency by efficiently managing GPU memory and task placement. The system aims to place tasks where data dependencies are satisfied, collocate tasks from the same job, and manage GPU memory effectively. By unifying these functions, Compass shows a significant reduction in completion times compared to other state-of-the-art schedulers while using the same or fewer resources. The system is designed to handle edge clusters physically close to end-users, targeting rapid response times for interactive applications utilizing machine intelligence.

Stats

Comparison with other state of the art schedulers shows a significant reduction in completion times. In one case, just half the servers were needed for processing the same workload. ML model parameters can be hundreds of megabytes in size. GPU memory is treated as a cache with the cache hit rate being an important metric.

Quotes

Key Insights Distilled From

Compass

by Yuting Yang,... at arxiv.org 02-29-2024

https://arxiv.org/pdf/2402.17652.pdf

Deeper Inquiries

How does Compass handle dynamic adjustments during task execution

Compass handles dynamic adjustments during task execution by continuously monitoring the system state and making real-time decisions based on changing conditions. When a task completes, Compass evaluates if any rescheduling is necessary due to updated information about worker loads, GPU cache contents, or model availability. This dynamic adjustment phase allows Compass to adapt to unforeseen changes in the system and optimize task assignments for reduced latency.

What are potential drawbacks of a fully decentralized scheduling approach like Compass

One potential drawback of a fully decentralized scheduling approach like Compass is the increased complexity of coordination among workers. With each worker capable of making independent scheduling decisions, there may be challenges in ensuring global optimization and avoiding conflicts between nodes. Additionally, decentralized systems can introduce overhead from frequent communication and synchronization requirements across all nodes, impacting overall efficiency.

How can Compass adapt to varying workloads and maintain optimal performance

Compass can adapt to varying workloads and maintain optimal performance through its intelligent decision-making algorithms and flexible architecture. By incorporating features like dynamic adjustment during task execution, eviction policies for GPU memory management, and prioritizing model locality awareness, Compass can efficiently allocate tasks based on current system states. This adaptability allows Compass to handle fluctuations in workload demands while optimizing resource utilization and minimizing job completion times.