In order to overcome the drawback of traditional method directly using image features to classify images, which will produce the problems of feature redundancy and low accuracy，a new approach based on Covariance Descriptor and LogitBoost was proposed for the image segmentation of traffic scene. The motion and structure , texture and HOG features were extracted for segmenting image. Meanwhile, the covariance descriptor was used to fuse the features mentioned above to reduce the feature redundancy.The multiclass LogitBoost classifier was used for image segmentation to improve the accuracy of segmentation. Experiments on the public CamVid dataset were preformed to test and evaluate the proposed method, and the results showed that this method was effective.