Heavy vehicle loads on urban bridges are one of the main reasons for the deterioration and damage of bridge performance. Traditional motion target detection algorithms suffer from a decrease in detection accuracy due to camera shake. This paper proposes a heavy vehicle recognition method based on the maximum structural similarity of images. A bridge field of view background model was established based on the time-domain median method， and the detected image and background model were divided into blocks. By searching for the maximum structural similarity near the corresponding block， this parameter is used as the basis for foreground/background classification to reduce the impact of camera shake. Fast Fourier transform algorithm was used to improve search speed for maximum structural similarity. Based on the vehicle’s outer envelope of a rectangular outer contour， the vehicle’s length， width， and height parameters were extracted， and sets of thresholds for heavy vehicle detection were set. The effectiveness of the proposed method was verified through the video of a certain elevated bridge. The results showed that， even with significant camera shake， the proposed method can still accurately identify heavy vehicles.