A multi-group and preemptable scheduling of cloud resource based on HTCondor

doi:10.1088/1742-6596/898/9/092051

	A multi-group and preemptable scheduling of cloud resource based on HTCondor
	Jiang XW(姜晓巍); Zou JH(邹佳恒); Cheng YD(程耀东); Shi JY(石京燕); Jiang, Xiaowei; Zou, Jiaheng; Cheng, Yaodong; Shi, Jingyan
刊名	Journal of Physics: Conference Series
	2017
卷号	898 期号:9 页码:092051
ISSN号	1742-6588
DOI	10.1088/1742-6596/898/9/092051
文献子类	Proceedings Paper
英文摘要	Due to the features of virtual machine-flexibility, easy controlling and various system environments, more and more fields utilize the virtualization technology to construct the distributed system with the virtual resources, also including high energy physics. This paper introduce a method used in high energy physics that supports multiple resource group and preemptable cloud resource scheduling, combining virtual machine with HTCondor (a batch system). It makes resource controlling more flexible and more efficient and makes resource scheduling independent of job scheduling. Firstly, the resources belong to different experiment-groups, and the type of user-groups mapping to resource-groups(same as experiment-group) is one-to-one or many-to-one. In order to make the confused group simply to be managed, we designed the permission controlling component to ensure that the different resource-groups can get the suitable jobs. Secondly, for the purpose of elastically allocating resources for suitable resource-group, it is necessary to schedule resources like scheduling jobs. So this paper designs the cloud resource scheduling to maintain a resource queue and allocate an appropriate amount of virtual resources to the request resource-group. Thirdly, in some kind of situations, because of the resource occupied for a long time, resources need to be preempted. This paper adds the preemption function for the resource scheduling that implement resource preemption based on the group priority. Additionally, the way to preempting is soft that when virtual resources are preempted, jobs will not be killed but also be held and rematched later. It is implemented with the help of HTCondor, storing the held job information in scheduler, releasing the job to idle status and doing second matcher. In IHEP (institute of high energy physics), we have built a batch system based on HTCondor with a virtual resources pool based on Openstack. And this paper will show some cases of experiment JUNO and LHAASO. The result indicates that multi-group and preemptable resource scheduling is efficient to support multi-group and soft preemption. Additionally, the permission controlling component has been used in the local computing cluster, supporting for experiment JUNO, CMS and LHAASO, and the scale will be expanded to more experiments at the first half year, including DYW, BES and so on. Its evidence that the permission controlling is efficient. © Published under licence by IOP Publishing Ltd.
电子版国际标准刊号	1742-6596
会议地点	San Francisco, CA, United states
会议日期	October 10, 2016 - October 14, 2016
语种	英语
内容类型	期刊论文
源URL	[http://ir.ihep.ac.cn/handle/311005/285447]
专题	高能物理研究所_计算中心
作者单位	Institute of High Energy Physics, Chinese Academy of Sciences, Beijing, China
推荐引用方式 GB/T 7714	Jiang XW,Zou JH,Cheng YD,et al. A multi-group and preemptable scheduling of cloud resource based on HTCondor[J]. Journal of Physics: Conference Series,2017,898(9):092051.
APA	姜晓巍.,邹佳恒.,程耀东.,石京燕.,Jiang, Xiaowei.,...&Shi, Jingyan.(2017).A multi-group and preemptable scheduling of cloud resource based on HTCondor.Journal of Physics: Conference Series,898(9),092051.
MLA	姜晓巍,et al."A multi-group and preemptable scheduling of cloud resource based on HTCondor".Journal of Physics: Conference Series 898.9(2017):092051.