To study the vibration of the automobile transmission, the third gear of the automobile transmission was taken as the investigated subject. MASTA was used to carry out the load spectrum analysis. To optimize the module and helical angle of the third gear, the capacity ratio and the size of the gear transmission system of the third gear were improved. The new modulus and helix angle were obtained by the optimum design. Considering the gear meshing stiffness, backlash, bearing clearance and transmission error, a bending-torsion coupled nonlinear vibration model with 6-degree-of-freedom was established by the lumped mass method. The vibration characteristics were analyzed in detail. In order to compare the vibration characteristics before and after optimization intuitively, a method of translating the time-domain signal into the frequency domain signal was proposed by Fast Fourier Transform. The comparison of the frequency spectrum shows that the vibration characteristics are improved obviously. In particular, the reduction of vibration in z axial reaches about 1/4. It provides a theoretical basis for the vibration optimization of the automobile transmission.