DALiuGE: A graph execution framework for harnessing the astronomical data deluge

CORC > 云南天文台 > 中国科学院云南天文台 > 其他

	DALiuGE: A graph execution framework for harnessing the astronomical data deluge
	Wu, C.1; Tobar, R.1; Vinsen, K.1; Wicenec, A.1; Pallot, D.1; Lao, B.2; Wang, R.3; An, T.2; Boulton, M.1; Cooper, I.1
刊名	Astronomy and Computing
	2017-07
卷号	20 页码:1-15
关键词	Dataflow Graph execution engine Data driven Square kilometre array Many-task computing
ISSN号	22131337
通讯作者	Wu, C.
产权排序	第5完成单位
英文摘要	The Data Activated Liu1Graph Engine – DALiuGE2– is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both datasets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry datasets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities.
学科主题	天文学
收录类别	EI
原文出处	http://www.sciencedirect.com/science/article/pii/S2213133716301214
语种	英语
内容类型	期刊论文
源URL	[http://ir.ynao.ac.cn/handle/114a53/10042]
专题	云南天文台_其他
作者单位	1.International Centre for Radio Astronomy Research (ICRAR), The University of Western Australia, M468, 35 Stirling Highway, Crawley, Perth, WA, 6009, Australia 2.Shanghai Astronomical Observatory, Shanghai, China 3.Oak Ridge National Laboratory, TN, 37831, United States 4.Kunming University of Science and Technology Kunming, Yunnan, China 5.Yunnan Observatories Chinese Academy of Sciences Kunming, Yunnan, China
推荐引用方式 GB/T 7714	Wu, C.,Tobar, R.,Vinsen, K.,et al. DALiuGE: A graph execution framework for harnessing the astronomical data deluge[J]. Astronomy and Computing,2017,20:1-15.
APA	Wu, C..,Tobar, R..,Vinsen, K..,Wicenec, A..,Pallot, D..,...&Wang F.(2017).DALiuGE: A graph execution framework for harnessing the astronomical data deluge.Astronomy and Computing,20,1-15.
MLA	Wu, C.,et al."DALiuGE: A graph execution framework for harnessing the astronomical data deluge".Astronomy and Computing 20(2017):1-15.