Abstract: To account for intensified vibration due to winding inter-turn short circuit of the turbine generator rotor, the coupled nonlinear dynamic model of oil film force and electromagnetic force was established, and the dynamic stability analysis method was proposed. An improved shooting method for solving the cycle solution of the unbalanced electromagnetic force bearing-rotor system was developed; and the system damping criterion in examining system stability margin was presented. The calculation on the nonlinear vibration for the 300 MW turbine generator was finished, and the vibration response characteristics and instability degree in a typical fault mode of winding inter-turn short circuit were obtained. The results indicated that the winding inter-turn short circuit fault might lead to the low frequency instability phenomenon, and the frequency was close to the first critical speed of the system. The results will provide technical support and theoretical basis for direct prediction of the winding inter-turn short-circuit fault via vibration detection.