The finite element method for strength reduction recently used in slope stability analysis has disadvantages in revelation of soil deformation behavior and instability criterion. Based on stress strain curves of soil

Duncan-Zhang nonlinear elastic model and direct reduction of strength parameters (cohesion c and friction angle φ) of soil

discount method for tangent modulus Et is presented. The connotation of reduction is entirely revealed. And the exchange of deformation energy in potential slip area is proposed as instability criterion. On the characteristics of relation curve of deformation energy in slip area and reduction coefficient

two straight lines are drawn

the value of reduction coefficient at intersection point of the two lines is choice of safety factor of slope stability. The calculation results by modified Duncan’s finite element method with new instability criterion and references are compared

the influence of calculation area on deformation energy is analyzed

the new method is used in slope stability analysis of south channel of East line in South-to-North Water Diversion Engineering. All results show that the soil behaviors of deformation and strength are more entirely considered in modified Duncan’s finite element method for shear strength reduction

new instability criterion of slope is reliable and calculation is steady; safety of slope construction can be ensured.