Automating Characterization Deployment in Distributed Data Stream Management Systems

doi:10.1109/TKDE.2017.2751606

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Automating Characterization Deployment in Distributed Data Stream Management Systems
	Wang, Chunkai 1; Meng, Xiaofeng 1; Guo, Qi 2; Weng, Zujian 1; Yang, Chen 1
刊名	IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
	2017-12-01
卷号	29 期号:12 页码:2669-2681
关键词	Stream processing system relational query system incremental learning modeling and prediction
ISSN号	1041-4347
DOI	10.1109/TKDE.2017.2751606
英文摘要	Distributed data stream management systems (DDSMS) are usually composed of upper layer relational query systems (RQS) and lower layer stream processing systems (SPS). When users submit new queries to RQS, a query planner needs to be converted into a directed acyclic graph (DAG) consisting of tasks which are running on SPS. Based on different query requests and data stream properties, SPS need to configure different deployments strategies. However, how to dynamically predict deployment configurations of SPS to ensure the processing throughput and low resource usage is a great challenge. This article presents OrientStream, a framework for automating characterization deployment in DDSMS using incremental machine learning techniques. By introducing the data-level, query plan-level, operator-level, and cluster-level's four-level feature extraction mechanism, we first use the different query workloads as training sets to predict the resource usage by DDSMS, and select the optimal resource configuration from candidate settings based on the current query requests and stream properties, then migrate the operator state by introducing dynamic reconfiguration. Finally, we validate our approach on the open source SPS-Storm. In view of the application scenarios with long monitoring cycle and non-frequent data fluctuation, experiments show that OrientStream can reduce CPU usage of 8-15 percent and memory usage of 38-48 percent, respectively.
资助项目	Natural Science Foundation of China[91646203] ; Natural Science Foundation of China[61379050] ; Natural Science Foundation of China[61532016] ; Natural Science Foundation of China[61532010] ; Natural Science Foundation of China[61762082] ; National Key Research and Development Program of China[2016YFB1000602] ; National Key Research and Development Program of China[2016YFB1000603] ; Renmin University[11XNL010] ; Science and Technology Opening up Cooperation project of Henan Province[172106000077]
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE COMPUTER SOC
WOS记录号	WOS:000414712700004
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/6490]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Meng, Xiaofeng
作者单位	1.Renmin Univ China, Sch Informat, Beijing 100872, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Wang, Chunkai,Meng, Xiaofeng,Guo, Qi,et al. Automating Characterization Deployment in Distributed Data Stream Management Systems[J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,2017,29(12):2669-2681.
APA	Wang, Chunkai,Meng, Xiaofeng,Guo, Qi,Weng, Zujian,&Yang, Chen.(2017).Automating Characterization Deployment in Distributed Data Stream Management Systems.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,29(12),2669-2681.
MLA	Wang, Chunkai,et al."Automating Characterization Deployment in Distributed Data Stream Management Systems".IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 29.12(2017):2669-2681.