An important factor of rich dynamics in the gear transmission system is that there are a large number of various types of co-existing attractors. When multiple attractors coexist, the change of motion conditions and the inevitable disturbance may cause the gear transmission system to jump between different motion behaviors. As a result, the whole machine is adversely affected, and sometimes, the system structure will be destroyed. At present, some hidden attractors have not been found, and the bifurcation evolution characteristics of coexisting attractors have not been fully revealed. A single-degree-of-freedom spur gear system is considered. The Poincaré mapping compounded by local maps is constructed, and semi-analytic calculation method of eigenvalues of Jacobi matrix is presented. The stability and bifurcations of coexisting attractors are studied by applying numerical simulation, continuation shooting method and Floquet multipliers, and the basins of attraction of coexisting attractors are calculated by using cell mapping method. The influence of the meshing frequency, damping ratio and amplitude of time-varying excitation on the system dynamics is analyzed, and the discontinuous bifurcation behaviors including PD-type grazing bifurcation, saddle-node bifurcation induced by subcritical period-doubling bifurcation and boundary crisis are revealed in the gear transmission system. The saddle-node bifurcation induced by period-doubling bifurcation leads to the jump and hysteresis in the transition between adjacent periodic attractors, resulting in that the period-doubling bifurcation presents subcritical feature. The saddle-node bifurcation is a major factor for the appearance and disappearance of coexisting periodic attractors. The boundary crisis leads the chaotic attractor and its basin of attraction to disappear, and the bifurcation of corresponding periodic attractor terminates.