Adaptive Deep Metric Learning for Affective Image Retrieval and Classification

doi:10.1109/TMM.2020.3001527

CORC > 自动化研究所 > 中国科学院自动化研究所 > 智能感知与计算研究中心

	Adaptive Deep Metric Learning for Affective Image Retrieval and Classification
	Yao, Xingxu 1; She, Dongyu 1; Zhang, Haiwei 1; Yang, Jufeng 1; Cheng, Ming-Ming 1; Wang, Liang 2
刊名	IEEE TRANSACTIONS ON MULTIMEDIA
	2021
卷号	23 页码:1640-1653
关键词	Measurement Visualization Semantics Feature extraction Task analysis Image analysis Image retrieval Affective image retrieval convolutional neural network deep metric learning visual sentiment analysis
ISSN号	1520-9210
DOI	10.1109/TMM.2020.3001527
通讯作者	Zhang, Haiwei(zhhaiwei@nankai.edu.cn)
英文摘要	An image is worth a thousand words. Many researchers have conducted extensive studies to understand visual emotions since an increasing number of users express emotions via images and videos online. However, most existing methods based on convolutional neural networks aim to retrieve and classify affective images in a discrete label space while ignoring both the hierarchical and complex nature of emotions. On the one hand, different from concrete and isolated object concepts (e.g., cat and dog), a hierarchical relationship exists among emotions. On the other hand, most widely used deep methods depend on the representation from fully connected layers, which lacks the essential texture information for recognizing emotions. In this work, we address the above problems via adaptive deep metric learning. Specifically, we design an adaptive sentiment similarity loss, which is able to embed affective images considering the emotion polarity and adaptively adjust the margin between different image pairs. To effectively distinguish affective images, we further propose the sentiment vector that captures the texture information extracted from multiple convolutional layers. Finally, we develop a unified multi-task deep framework to simultaneously optimize both retrieval and classification goals. Extensive and thorough evaluations on four benchmark datasets demonstrate that the proposed framework performs favorably against the state-of-the-art methods.
资助项目	Major Project for New Generation of AI[2018AAA0100403] ; NSFC[61876094] ; NSFC[U1933114] ; Natural Science Foundation of Tianjin, China[18JCYBJC15400] ; Natural Science Foundation of Tianjin, China[18ZXZNGX00110] ; Open Project Program of the National Laboratory of Pattern Recognition (NLPR) ; Fundamental Research Funds for the Central Universities
WOS关键词	REPRESENTATIONS ; SIMILARITY ; FEATURES
WOS研究方向	Computer Science ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000655830300013
资助机构	Major Project for New Generation of AI ; NSFC ; Natural Science Foundation of Tianjin, China ; Open Project Program of the National Laboratory of Pattern Recognition (NLPR) ; Fundamental Research Funds for the Central Universities
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/45319]
专题	自动化研究所_智能感知与计算研究中心
通讯作者	Zhang, Haiwei
作者单位	1.Nankai Univ, Coll Comp Sci, Tianjin 300350, Peoples R China 2.Chinese Acad Sci, Inst Automat, CAS Ctr Excellence Brain Sci & Intelligence Techn, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Yao, Xingxu,She, Dongyu,Zhang, Haiwei,et al. Adaptive Deep Metric Learning for Affective Image Retrieval and Classification[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2021,23:1640-1653.
APA	Yao, Xingxu,She, Dongyu,Zhang, Haiwei,Yang, Jufeng,Cheng, Ming-Ming,&Wang, Liang.(2021).Adaptive Deep Metric Learning for Affective Image Retrieval and Classification.IEEE TRANSACTIONS ON MULTIMEDIA,23,1640-1653.
MLA	Yao, Xingxu,et al."Adaptive Deep Metric Learning for Affective Image Retrieval and Classification".IEEE TRANSACTIONS ON MULTIMEDIA 23(2021):1640-1653.