Abstract: Herringbone gears have strong bearing capacity, large contact ratio and high reliability, and are mostly used in high-speed and heavy-load working occasions. Exploring the nonlinear dynamic characteristics of high-speed and heavy-duty herringbone gear transmission system and finding out the stable operation interval of the system can provide reference for its design. Firstly, the time-varying meshing stiffness of gear pair is calculated, and the time-varying meshing force is calculated by introducing the backlash, the nonlinear function of backlash and the comprehensive transmission error. The bearing clearance is introduced to calculate the bearing force. Subsequently, the nonlinear dynamic equation of the high-speed and heavy-duty herringbone gear transmission system is established, and the fourth-order Runge-Kutta method is used to solve the equation. Finally, the influence of different factors on the stability of the system is explored. Keeping other parameters of the system unchanged, the meshing damping, backlash, meshing stiffness and excitation frequency are changed respectively. The time-displacement image, time-velocity image, spatial phase diagram, spatial frequency diagram and bifurcation diagram of the system are drawn. The change trend of nonlinear dynamic response of the system is observed and the motion state of the system is judged. The results show that within a certain range, the stability of the system is positively correlated with meshing damping and meshing stiffness, and negatively correlated with the backlash. When the external excitation frequency is gradually increased, the system motion gradually changes from single-cycle motion to chaotic motion, and then returns to stable single-cycle motion. Therefore, in order to ensure the smooth operation of the system, the external excitation frequency should be reasonably selected.