On the basis of the fundamental damage mechanism of concrete, damage accumulation and residual deformation of concrete under fatigue loading were considered, and the constitutive equation of concrete expressed by Helmholtz free energy was derived. An energy based fatigue damage constitutive model was then proposed by using the variation principle. Meanwhile, based on the development law of the residual deformation of concrete under fatigue load, the influencing coefficient of the residual deformation with definite physical meaning was defined, which was then included into the plastic damage constitutive model of concrete together with the damage factor of deformation modulus in order to provide a more accurate and convenient fatigue constitutive model. To verify the accuracy and applicability of the proposed model, the results of the numerical examples were compared with the experimental data, where no more than 3% of error was observed.