For a metal gear pair in the core parts of planetary reducer with few teeth difference, the problems of poor deformation compensation can be solved by designing a metal rubber complex gear pair, which reduces the vibration and achieves deformation compensation. The simulation analysis with rigid flexible coupling multi-body dynamics was used to verify the vibration of the metal rubber complex gear and metal gear pair. The vibration characteristics of these two gear pairs under different operating conditions were also studied by the experimental method. The results of simulation and experiment show that the vibration characteristics of the complex gear pair are improved by the addition of the metal rubber material, and the complex gear pair is obviously superior to the metal gear pair in meshing force, angular acceleration, transmission efficiency and vibration acceleration. The transmission stability of the complex gear pair is also greatly improved. Moreover, the simulation results are consistent with the experimental ones.