Abstract: To address nonlinear vibration of high-speed flexible rotor systems near critical speeds, a nonlinear dynamic model of the flexible rotor is established using the three-dimensional absolute nodal coordinate formulation (ANCF). A three-dimensional two-node beam element with reduced gradients is employed to discretize the shaft, and the disk is simplified as a lumped mass with rotational inertia. The nonlinear dynamic equations are derived via Euler-Lagrange equation. To verify the theoretical model, the obtained results are compared with the finite element method. It shows that when the rotational speed is 100 rad/s, the contrast error for the first-order precession frequency is only 0.17%, and for the forward/reverse precession frequencies of the first six modes is within 5%. The predicted vibration amplitude is further compared with previous experimental data, and the maximum amplitude comparison error is within 10%. Based on the theoretical model established by ANCF, the linear and nonlinear dynamic characteristics of the flexible rotor are studied. Linear dynamic analysis shows that as the rotational speed increases, the difference in the forward and reverse precession frequencies of the flexible rotor becomes increasingly significant, indicating that the gyroscopic effect of the rotor has a more significant impact on the dynamic characteristics at high rotational speeds. Nonlinear dynamic analysis indicates that near the first-order critical speed, the rotor exhibits single-period vibrations at the same frequency as the rotational frequency and shows a hardening characteristic behavior. However, when the rotational speed increases to the vicinity of the second-order critical speed, the rotor evolves from single-period to quasi-periodic vibrations and exhibits a superharmonic resonance behavior. The rotor vibration frequency not only contains the second-order frequency but also the second harmonic component of the first-order frequency. The vibration shape of the rotor shows the superposition feature of the first and second-order modes. This nonlinear vibration behavior arises from modal coupling induced by geometric nonlinearity, which causes energy to be directionally transferred from higher-order modes to lower-order modes and manifests as superharmonic responses. Additionally, the stiffening effect of centrifugal stiffness shifts the frequency corresponding to the resonance peak.