Abstract: As a kind of good damping material, viscoelastic material is widely used in machinery, aviation, civil engineering and other fields. A new viscoelastic nonlinear energy sink is proposed by replacing the damping element with a viscoelastic Maxwell element in the traditional nonlinear energy sink, and the vibration damping performance of the model under harmonic excitation is investigated. Firstly, the dynamic equation of the system is established according to Newton's second law. The harmonic balance method is adopted to obtain the system amplitude-frequency response curves, and the Runge-Kutta numerical method in MATLAB is applied to verify the correctness of the analytical solution, and it is found that the results are in good agreement. Then, the damping performance of the viscoelastic nonlinear energy sink and the influence of parameters are analyzed. Finally, the variation trend of vibration reduction effect are analyzed when the nonlinear stiffness ratio and damping ratio change at the same time under different mass ratios, and the optimal value range of viscoelastic nonlinear energy sink is discussed. The results show that the maximum amplitude of the primary system first decreases and then increases with the increase of nonlinear stiffness. When the parameters are selected properly, the damping effect of viscoelastic nonlinear energy sink is better than that of traditional nonlinear energy sink. In addition, with the increase of the mass ratio, the minimum value of the maximum amplitude of the primary system first decreases and then tends to remain unchanged, and the optimal range of nonlinear stiffness ratio and damping ratio is greater. The above conclusions provide a theoretical basis for the practical application of the viscoelastic nonlinear energy sink.