Long-term tracking is one of the most challenging problems in computer vision. In this paper, we make full use of the Discriminative Correlation Filter (DCF), and propose a real-time long-term tracker by exploiting a joint tracking–verification–detection–refinement framework. We utilize a DCF which is updated aggressively to estimate translation and scale variation of the target. Subsequently, a passively updated DCF checks the reliability of the tracking result. Once the result is not reliable, we evoke the proposed optimized candidate detector to generate a small number of relatively high quality candidates. Finally, one DCF with an adaptive online learning rate is adopted to refine the predictions that the sparse candidates inferred. In addition, we employ a selection mechanism for the correlation responses to maintain reliable samples effectively. Extensive experiments show that the proposed method performs favorably against lots of state-of-the-art methods while running more than 30 frames per second on single CPU. © 2020 Elsevier Inc.