There still remains some problems which have not been solved in RPN-based trackers, including data imbalance, inappropriate proposals and poor robustness to spatial rotation even scale variation. We propose a cascaded region proposal network framework for visual tracking based on region proposal networks, spatial transformer networks and proposal selection strategy. We first to extract the features from deep and shallow layers via cascaded region proposal network to ensure the spatial information and semantic cue of the appearance model. Then, the feature extraction model based on spatial transformer networks is performed to calculate the parameters of spatial transformer and obtain the fused features. During the tracking and testing of proposed networks, the proposals are generated and re-ranked by formulating the proposals selection strategy to ensure the localization and scale of the estimated target. We extensively prove the effectiveness of the proposed method though the ablation studies of the tracking benchmark which include OTB2015, VOT2016 and UAV123. The experimental results perform that the accuracy and robustness of the proposed method as the real-time tracker and the long-term tracker as well. In the meantime, the test on the benchmark UAV123 shows that the tracker can be employed to some engineering area. © Springer Nature Switzerland AG 2020.