The heavy vehicle load on urban bridges is one of the main reasons for the deterioration and damage of bridge performance. Traditional motion target detection algorithms suffer from a decrease in detection accuracy due to camera shake. This article proposes a heavy vehicle recognition method based on the maximum structural similarity of images. A bridge field of view background model was established based on the time-domain median method, and the detected image and background model were divided into blocks. By searching for the maximum structural similarity near the corresponding block, this parameter is used as the basis for foreground/background classification to reduce the impact of camera shake. Fast Fourier transform algorithm was used to improve search speed for maximum structural similarity. Fitting the vehicle's outer envelope with a rectangular outer contour, extracting the vehicle's length, width, and height parameters, and setting sets of thresholds for heavy vehicle detection. The effectiveness of the proposed method was verified through the video of a certain elevated bridge. The results showed that even with significant camera shake, the proposed method can still accurately identify heavy vehicles.