It is difficult for the computer to distinguish the target from the background due to the long-time static of the target after moving. A new moving target detection and background reconstruction algorithm is proposed and is applied into the RGB video for the first time. Firstly, the proposed algorithm builds a model from the time dimension to extract the changed region. Then, it combines with the space dimension information to completely extract the moving target. The spatiotemporal correlation model is established to realize the construction of pure background. The experimental results show that the proposed algorithm can effectively reconstruct the background and the recognition rate of moving target is high. © 2020, Springer-Verlag GmbH Germany, part of Springer Nature.