It is difficult to detect the foreground targets accurately in the sequence images in the situation of moving background. According to the characteristics of the targets in moving backgrounds, a fast visual saliency-aided ViBe method was proposed. The “memory window” random sample consensus algorithm was first proposed to estimate the background motion model. Then, the compensated frames were delivered into the detection algorithm with adaptive background updating factor determined by the two-dimensional entropy and saliency of the image. The saliency was also used to suppress the “ghost” effect. Experimental results show that, in the absence of prior knowledge of the moving foreground targets, the proposed method can effectively detect targets in the situation of moving background, and it also has good robustness and high efficiency.