A principle of nonlinear relationship proposed by Hoek is elaborated

then based on that

a new algorithm of obtaining shear strength of rock mass is brought forward by considering the overall or average level when rock mass is broken. Through the new algorithm

the magnitudes of shear stress can be calculated at different given normal stresses

and a series of shear stress and normal stress are generated; then the values of cohesion and internal frictional angle are determined via linear regression analysis

which solves the determination of shear strength under certain normal stress that Hoek-Brown criterion fitting algorithm can not do. Then a research on shear strength of rock mass from China-Myanmar oil and gas pipelines (domestic section) Lancang River across domain engineering as an example and a further comparison study is taken. It is shown that the values of cohesion and internal frictional angle determined by the first method according to the new algorithm are little different to these gained through the Hoek-Brown criterion fitting algorithm. The reason is that the factors considered are not the same

while the normal stress is in considered by the two algorithms resulting in confining pressure and normal stress are varied practical engineering is low

the cohesion and internal friction angle proposed by the second method according to the new algorithm

the results obtained are close to the actural situation

which has overcome the problem that Hoek-Brown criterion fitting algorithm relies on the confining pressure.