In order to set up a new detection method for the compactness of highway subgrade, the subgrade deformation and mechanics mechanism under static penetration were discussed with the nonlinear variation characteristics of the subgrade soil deformation. The idea with step-loading and the layer-wise summation analysis of ground foundation settlement was introduced with the Duncan-Chang model and the incremental generalized Hooke's law. Therefore, an analysis model between the subgrade penetration load and its displacement with the parameters about initial porosity ratio, Poisson ratio, cohesive strength and angle of internal friction was developed. On the basis of the static penetration displacement analysis model mentioned above, a method of back analysis based on the adaptive genetic simulated-annealing algorithm was recommended according to the subgrade static penetration load and displacement test curves, then a new detection technique for the compactness of subgrade based on static penetration was set up, which can not only detect the compactness of subgrade, but also check the other physical and mechanical parameters of the subgrade soil. And it still has the features with fast and efficient detecting. Moreover, by comparison with the other existing methods and the computation of an engineering example, the proposed method was proved to be reasonable and feasible.