Abstract: According to the large error problem of modal analysis of the rotor system for magnetic levitation motors, the nonlinear contact behavior between rotor components was considered. The flexible rotor system modal can be analyzed accurately based on the penalty method by optimizing the contact stiffness factor. Through the establishment of the rotor-bearing system finite element analysis model, the ten order natural frequencies and variations of the rotor system under different bearing stiffness were obtained. The simulation results are in good agreement with the test values. This article also analyzed the rotor system with the dynamic balancing ring. The accuracy of results has been proved by the critical speed tests of the experimental prototype.