To extract the damage sensitive features directly from the vibration response, a damage identification method based on the wavelet packet decomposition-local mean decomposition and the permutation entropy was proposed, without requiring the exciting force. The wavelet packet decomposition was applied to decompose the original vibration signal into a set of narrow band signals, and then, the local mean decomposition method was used to decompose the narrow band signal, which can extract the low energy components effectively. After that, the permutation entropy was obtained to detect the structural damage. The relative permutation entropy between the test data and the sample data was obtained to identify the damage pattern. The experiment results show that the proposed method can be used to identify the structural damage effectively.