A template matchbased method was presented by using the contours of image sequence as representative descriptors of human posture to achieve human action recognition. First, the entire binary image of human silhouette was obtained by background subtraction and shadow elimination from video sequences. A new contour descriptor was defined, which was employed to transform the contour into a 1D distance signal. An approximating polygon was constructed by connecting the neighbor segmenting points by using line segment after equally segmenting the contour and the centroid of silhouette was obtained. The three distance strings formed with the distance from the vertexes to the center of contour and the distances from the vertexes to the two subsequence vertexes were used to describe the contour. A hierarchical clustering method was proposed to extract the key postures of human action and a human behavior sequence could be presented by a character string if a key posture was presented by a character. The edit distance was proposed to measure the similarity of the templates and the test sequences. Experiment results have shown that this method can achieve a correct recognition rate of above 85.7 %.