Abstract: A modified Rayleigh oscillator with multiplelimit cycles is investigated by means of a new analytical method fornonlinear problems, namely, the homotopy analysis method (HAM). TheHAM is independent upon small parameters. More importantly, unlike other traditional techniques, the HAM provides us with asimple way to ensure the convergence of solution series. Thus, theHAM can be used for strongly nonlinear problems. Comparisons of thesolutions given by the HAM, the method of averaging, and Runge-Kuttamethod show that the method of averaging is not valid for stronglynonlinear cases, and the Runge-Kutta numerical technique does notwork for the instable limit cycles,however, the HAM not only works for strongly nonlinear cases, butalso can give good approximations for the instable limit cycles.