To realize the tomato stem classification of irregular thin and long objects with a similar color to the background， an algorithm for tomato stem classification based on improved Mask R-CNN was proposed. First， the images of daytime and night tomato plants were collected， and stems in the images were labeled using the Labelme to produce data sets of tomato stem classification at daytime and night， separately. Then， the Mask R-CNN model was trained separately using the two data sets with the transfer learning method. The Mask branch was improved to generate its minimum external moment when the Mask was produced， an evaluation index Re was proposed to evaluate the accuracy of the Mask border， and pixel-level evaluation indexes were proposed to comprehensively evaluate the performance of the model. Test results show that the pixel F1 score and pixel mean average precision of the night and day stem classification model are 48.82%， 50.03%， and 57.76%， 56.06%， respectively. After the improvement of the Mask branch， the accuracy of the mask border is significantly improved. The average recognition time of each image is 0.31 s， indicating that the recognition model can meet the real-time requirements of the algorithm in practical applications. It can provide method support for the intelligence of plant pruning and other works.