The nonlinear failure criterion is employed to determine the active earth pressure and the position of slip surface in this paper. Firstly

the variables and are created by tangents method

then and corresponding to different potential slip surfaces are calculated by iteration method and the active earth pressure is obtained by using generalized Coulomb soil pressure theory. The slip surface

which has the maximum active earth pressure

should be the most dangerous slip surface

and the maximum active earth pressure is also what we desired. After comparing with the research results by Mohr-Coulomb linear failure criterion

it is concluded that the active earth pressures behind the rigid retaining walls

which are calculated by the nonlinear failure criterion

will be more practical and accurate

while adopting the Mohr-Coulomb linear failure criterion will provide a lower active earth pressure values

therefore

will cause insecurity in real project.