The main problems in the detection of shield tunnel lining defects are how to obtain high quality images of different defects and how to quickly and accurately detect the defects. A device for mobile tunnel inspection (MTI-100) was designed and manufactured based on CCD line array cameras. Using MTI-100, Shanghai Metro Lines 1, 2, 4, 7, 8, 10 and 12 were tested and a large number of lining images were obtained. These images were manually labeled to form a high quality database of lining defects samples. Based on the Faster R-CNN (Faster Region-based Convolutional Neural Network), a deep learning framework for automatic disease detection was established. Inspired by the existing model VGG16, the CNN detection model of tunnel lining defects was established. Considering the particularity of cracks and leakage defects, statistical analysis and K-means clustering algorithm were used to analyze the geometric features, so as to optimize the related parameters of anchor box in the VGG-16 network model. The results show that the accuracy of the optimizeation is greatly improved（about 7%），and the training time is reduced. It is verified that the method can also improve the accuracy of the defect detection model of crack or leakage singly.