Deep Fusion of Multiple Semantic Cues for Complex Event Recognition

doi:10.1109/TIP.2015.2511585

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Deep Fusion of Multiple Semantic Cues for Complex Event Recognition
	Zhang, Xishan 1,6; Zhang, Hanwang 2; Zhang, Yongdong 6; Yang, Yang 3; Wang, Meng 4; Luan, Huanbo 5; Li, Jintao 6; Chua, Tat-Seng 2
刊名	IEEE TRANSACTIONS ON IMAGE PROCESSING
	2016-03-01
卷号	25 期号:3 页码:1033-1046
关键词	Multimedia event recognition deep learning fusion
ISSN号	1057-7149
DOI	10.1109/TIP.2015.2511585
英文摘要	We present a deep learning strategy to fuse multiple semantic cues for complex event recognition. In particular, we tackle the recognition task by answering how to jointly analyze human actions (who is doing what), objects (what), and scenes (where). First, each type of semantic features (e.g., human action trajectories) is fed into a corresponding multi-layer feature abstraction pathway, followed by a fusion layer connecting all the different pathways. Second, the correlations of how the semantic cues interacting with each other are learned in an unsupervised cross-modality autoencoder fashion. Finally, by fine-tuning a large-margin objective deployed on this deep architecture, we are able to answer the question on how the semantic cues of who, what, and where compose a complex event. As compared with the traditional feature fusion methods (e.g., various early or late strategies), our method jointly learns the essential higher level features that are most effective for fusion and recognition. We perform extensive experiments on two real-world complex event video benchmarks, MED'11 and CCV, and demonstrate that our method outperforms the best published results by 21% and 11%, respectively, on an event recognition task.
资助项目	National High Technology Research and Development Program of China[2014AA015202] ; National University of Singapore-Tsinghua Extreme Search Project[R-252-300-001-490] ; National Nature Science Foundation of China[61525206] ; National Nature Science Foundation of China[61428207] ; National Nature Science Foundation of China[61303075]
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000378293900002
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/8386]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Yongdong
作者单位	1.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 2.Natl Univ Singapore, Sch Comp, Singapore 117417, Singapore 3.Univ Elect Sci & Technol China, Sch Comp Sci & Engn, Chengdu 611731, Peoples R China 4.Hefei Univ Technol, Sch Comp Sci & Informat Engn, Hefei 230009, Peoples R China 5.Tsinghua Univ, Dept Comp Sci & Technol, Beijing 100084, Peoples R China 6.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Xishan,Zhang, Hanwang,Zhang, Yongdong,et al. Deep Fusion of Multiple Semantic Cues for Complex Event Recognition[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2016,25(3):1033-1046.
APA	Zhang, Xishan.,Zhang, Hanwang.,Zhang, Yongdong.,Yang, Yang.,Wang, Meng.,...&Chua, Tat-Seng.(2016).Deep Fusion of Multiple Semantic Cues for Complex Event Recognition.IEEE TRANSACTIONS ON IMAGE PROCESSING,25(3),1033-1046.
MLA	Zhang, Xishan,et al."Deep Fusion of Multiple Semantic Cues for Complex Event Recognition".IEEE TRANSACTIONS ON IMAGE PROCESSING 25.3(2016):1033-1046.