Abstract: Fractional calculus has many excellent characteristics and is mainly used to improve the research accuracy for vibration characteristics of nonlinear systems in the field of dynamics. In this paper, the fractional-order derivative is introduced into the quasi-periodic Mathieu equation and the influences of fractional-order term on the stability of the equation are studied. Firstly, the conditions of the periodic solutions are obtained by the perturbation method, and the approximate expressions of the transition curves are also gotten. The accuracy of the approximate analytical solution is verified by comparing with the numerical solution, and they are in good agreement with each other. Moreover, approximate expressions of transition curves under different conditions are summarized. By analyzing their formal characteristics, it is found that the fractional-order term exists in the form of equivalent linear stiffness and equivalent linear damping in the equation, the general forms of equivalent linear damping and equivalent linear stiffness are obtained, and the thickness of unstable region is defined. Finally, the effects of fractional-order parameters on the size of stability region and the position of transition curves are analyzed intuitively by numerical method. It is found that the fractional-order term has both damping and stiffness characteristics, and the fractional coefficient and fractional order affect the transition curves of the equation in the form of equivalent linear stiffness and equivalent linear damping. Even in some cases, the effect of fractional-order term is almost equal to linear damping or linear stiffness. Reasonable selection of fractional-order parameters can make it show different degrees of stiffness or damping characteristics, and have different degrees of influence on the stability region of the equation and the position of the transition curve, thus affecting the value range of the stability parameters of the equation. These results are of great significance for the study of dynamic characteristics of such systems.