A new method based on optimized TLD (Track-Learning-Detection) and SVM (Support Vector Machine) for tracking pedestrian was proposed. First, with pedestrians as positive samples and the background as negative samples respectively, HOG (Histogram of Oriented Gradient) descriptor of pedestrian was extracted and combined with linear SVM to train the pedestrian classifier，which was used to obtain the calibrated pedestrian area accurately. Then, adaptive tracking and online learning on the pedestrians on the basis of TLD were integrated to estimate the reliability of the positive and negative samples, to rectify error existing in the current frame caused by detection and to update the tracking data simultaneously to avoid subsequent similar mistakes. The experiment results demonstrate that, compared with the conventional tracking algorithm, the proposed algorithm can not only significantly adapt to occlusions and appearance changes but also automatically identify and track pedestrian targets at arbitrary position, manifesting stronger robustness.