This paper investigates the traffic sign recognition task with deep learning methods. The proposed algorithm which is called DeepSign includes three modules: a detection module (PosNet) for locating the traffic sign in a static image, a classification module (PatchNet) for classifying the detected image patch, and a temporal filter for correcting the recognition results. The PosNet is a binary object detection convolution neural network which regards all traffic signs as one class and the background as the other class. Different from the traditional works which recognize the traffic sign on the static image, the proposed temporal filter exploits the contextual information to recover the missed detection region and correct the false classification. The experiments validate the effectiveness of the proposed algorithm. It achieved the third place on the traffic sign recognition task in 2017 China intelligent vehicle future challenge (2017 CIVFC).