Hierarchical Scheduling for Diverse Datacenter Workloads

TitleHierarchical Scheduling for Diverse Datacenter Workloads
Publication TypeConference Paper
Year of Publication2013
AuthorsBhattacharya, A. A., Culler D. E., Friedman E., Ghodsi A., Shenker S. J., & Stoica I.
Other Numbers3796

There has been a recent industrial effort to develop multi-resource hierarchical schedulers. However, the existing implementations have some shortcomings in that they might leave resources unallocated or starve certain jobs. This is because the multi-resource setting introduces new challenges for hierarchical scheduling policies. We provide an algorithm, which we implement in Hadoop, that generalizes the most commonly used multi-resource scheduler, DRF [1], to support hierarchies. Our evaluation shows that our proposed algorithm, H-DRF, avoids the starvation and resource inefficiencies of the existing open-source schedulers and outperforms slot scheduling.

Bibliographic Notes

Proceedings of the ACM Symposium on Cloud Computing, Santa Clara, California

Abbreviated Authors

A. A. Bhattacharya, D. Culler, E. Friedman, A. Ghodsi, S. Shenker, and I. Stoica

ICSI Research Group

Research Initiatives

ICSI Publication Type

Article in conference proceedings