In video surveillance systems, noise and shake of the background caused by complex environment can greatly influence the detection of moving objects. In order to solve this problem, a Gaussian Mixture Model (GMM) based on subzone gray projection video stabilization algorithm was proposed. First, each frame was divided into several blocks, and the subzone gray projection algorithm was applied to the frame to exactly extract the motion vector of each subzone and analyze the correlation between them. Based on the above analysis, we could decide whether a frame was with dithering or not, and make motion compensation for dithering frame. Then, we used GMM algorithm to extract the moving objects. Finally, morphology was applied as post-processing to further improve the detection accuracy. The subjective and objective evaluations of many different videos were implemented to verify the validity of the proposed algorithm in our experiments. The experiment results have shown that the proposed algorithm can detect the moving targets accurately from the scenes with dithering and suppress the false alarm rate significantly.