Searching critical slip surface is very important in slope stability analysis. For the deficiencies of the existing methods

a new search method is put forward. On the basis of global analysis method of slope stability

a nonlinear optimum model for determining critical sliding surface is proposed for calculating safety factor. In this method

both the safety factor and vertical coordinates of discrete points on the slip surface are taken as independent variables

and the safety factor is taken as the objective function. In the model

the equilibrium equations and convexity of the slip surface act as constraint conditions. Because the objective function is linear

and the constraint functions are polynomials of at most quadratic

the model has weaker nonlinearity and so it can be solved easily by those conventional nonlinear optimization tools

such as Matlab. The following two example results show that the proposed method has advantages for numerical stability and convergence.