Abstract: The planetary gear wear leads to the nonlinear increase of gear tooth clearance, decrease of the transmission accuracy, and increase of the tooth surface impact force, which will cause the aggravation of the vibration of the gear transmission system. Therefore, it is necessary to analyze the tooth surface wear and dynamic characteristics of planetary gears. In this paper, the nonlinear dynamic coupling calculation model of gear nonlinear wear and gear backlash is established and the gear wear characteristics of the planetary transmission is investigated. Firstly, the nonlinear dynamic model of gear meshing is established. And then, the nonlinear meshing force of gear motion is obtained. Secondly, the wear distribution law of gear tooth surface is presented by combining the nonlinear meshing force with the wear model of gear tooth surface. Further, the tooth surface is reconstructed according to the tooth clearance after wear, and the dynamics model of the gear is updated. Finally, the trend of dynamic meshing force and wear characteristics of planetary gear transmission can be obtained. The gear tooth vibration responses of the gear transmission system are presented. The numerical calculation results show that the increases of the planetary gear wear mainly affect the gear force in the alternating single-double tooth meshing. Moreover, the sun-planetary meshing tooth is more sensitive to wear, and the meshing condition of the tooth surface deteriorates greatly, which is the main reason causing the deterioration of planetary gear transmission. The presented work provides a theoretical basis for the working performance evaluation and reliability prediction of planetary gear transmission system.