Abstract: The dynamic evolution of tooth surface morphology induced by wear significantly alters meshing characteristics, where variations in time-varying mesh stiffness (TVMS) and transmission error serve as critical contributors to gear system failures. Conventional analytical methods often neglect the influence of wear on friction behavior, making it difficult to accurately reflect the dynamic evolution patterns under actual operating conditions. This study employs a modified Archard wear model to dynamically predict tooth surface wear in helical gears, establishing computational models for TVMS and transmission error that account for wear progression. By introducing the load distribution factor (LSF), a potential relationship between wear and friction is established. A friction coefficient model for helical gears under mixed lubrication conditions is proposed, incorporating variations in the equivalent radius along the contact line, along with a corresponding solution method. Based on a friction dynamics model of a helical gear-rotor-bearing system, the influence of tooth surface wear on system dynamic behavior under mixed lubrication conditions is investigated. The results indicate that at a rotational speed of 5818 rpm, vibration displacement increases with wear severity, transitioning from single-period to period-doubling motion. The amplitude fluctuates over time, with the amplitudes at the meshing frequency (f_m) and its third harmonic (3f_m) gradually increasing. Sidebands with intervals of 1/2f_m to 1/3f_m also emerge. As tooth surface wear intensifies and the rotational speed increases from 5200 rpm to 6400 rpm, the system exhibits a typical nonlinear evolution path, transitioning from chaotic motion through period-doubling bifurcation toward periodic motion. The sideband intervals gradually converge from 1/4f_m to f_m, and the influence of chaotic motion on vibration displacement surpasses that of wear. Consequently, the dominant mechanism of vibration amplification varies with speed ranges: in non-chaotic regions, it is primarily driven by wear-induced TVMS attenuation, increased transmission error, and friction coefficient variations; whereas in chaotic regions, it is dominated by chaotic nonlinear dynamics.