Aiming at the problems of poor identification accuracy in small sample size asphalt pavement damage classification recognition， five common types of pavement damage were selected. The shallow depth convolutional neural network model based on VGG is designed as an automatic classification method of asphalt pavement damage image. Firstly， the collected image samples are made into data sets for model training. Moreover， three different batch sizes and two different network layers are set up for training， and the most suitable size for the network model is selected so as to obtain the shallow VGG. The processed road image is directly used as the input of the model for training， verification and testing of the model. Finally， the test result was compared with SVM and the current mainstream deep convolutional neural network. The results show that the classification accuracy of the training set， verification set and a test set of shallow VGG is close， the classification and recognition accuracy rate of pavement damage image is more than 98%， which shows the good ability of recognition of shallow VGG. Compared with SVM and the current mainstream deep convolutional neural network， shallow VGG network model takes less time and has strong generalization ability， and can capture more global information. It can be seen that the shallow VGG model has significant advantages in the classification and recognition of small-scale images， it is more robust and the results are more accurate compared with other methods.