Spatio-Temporal Memory Attention for Image Captioning | |
Ji, Junzhong1,2; Xu, Cheng1,2; Zhang, Xiaodan1,2; Wang, Boyue1,2; Song, Xinhang3 | |
刊名 | IEEE TRANSACTIONS ON IMAGE PROCESSING |
2020 | |
卷号 | 29页码:7615-7628 |
关键词 | Image captioning spatio-temporal relationship attention transmission memory attention LSTM |
ISSN号 | 1057-7149 |
DOI | 10.1109/TIP.2020.3004729 |
英文摘要 | Visual attention has been successfully applied in image captioning to selectively incorporate the most relevant areas to the language generation procedure. However, the attention in current image captioning methods is only guided by the hidden state of language model, e.g. LSTM (Long-Short Term Memory), indirectly and implicitly, and thus the attended areas are weakly relevant at different time steps. Besides the spatial relationship of attention areas, the temporal relationship in attention is crucial for image captioning according to the attention transmission mechanism of human vision. In this paper, we propose a new spatio-temporal memory attention (STMA) model to learn the spatio-temporal relationship in attention for image captioning. The STMA introduces the memory mechanism to the attention model through a tailored LSTM, where the new cell is used to memorize and propagate the attention information, and the output gate is used to generate attention weights. The attention in STMA transmits with memory adaptively and dependently, which builds strong temporal connections of attentions and learns the spatio-temporal relationship of attended areas simultaneously. Besides, the proposed STMA is flexible to combine with attention-based image captioning frameworks. Experiments on MS COCO dataset demonstrate the superiority of the proposed STMA model in exploring the spatio-temporal relationship in attention and improving the current attention-based image captioning. |
资助项目 | National Natural Science Foundation of China[61906007] ; National Natural Science Foundation of China[61672065] ; National Natural Science Foundation of China[61906011] ; National Natural Science Foundation of China[61902378] ; Beijing Municipal Science and Technology Project[KM202010005014] |
WOS研究方向 | Computer Science ; Engineering |
语种 | 英语 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
WOS记录号 | WOS:000553851400028 |
内容类型 | 期刊论文 |
源URL | [http://119.78.100.204/handle/2XEOYT63/15883] |
专题 | 中国科学院计算技术研究所期刊论文_英文 |
通讯作者 | Zhang, Xiaodan |
作者单位 | 1.Beijing Univ Technol, Beijing Artificial Intelligence Inst, Beijing 100124, Peoples R China 2.Beijing Univ Technol, Fac Informat Technol, Beijing Municipal Key Lab Multimedia & Intelligen, Beijing 100124, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, CAS, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China |
推荐引用方式 GB/T 7714 | Ji, Junzhong,Xu, Cheng,Zhang, Xiaodan,et al. Spatio-Temporal Memory Attention for Image Captioning[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2020,29:7615-7628. |
APA | Ji, Junzhong,Xu, Cheng,Zhang, Xiaodan,Wang, Boyue,&Song, Xinhang.(2020).Spatio-Temporal Memory Attention for Image Captioning.IEEE TRANSACTIONS ON IMAGE PROCESSING,29,7615-7628. |
MLA | Ji, Junzhong,et al."Spatio-Temporal Memory Attention for Image Captioning".IEEE TRANSACTIONS ON IMAGE PROCESSING 29(2020):7615-7628. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论