Based on the transversely isotropic elastic constitutive model in the local coordinate system of bedding plane

according to the occurrence of layer

the stress conversion method from the global to local coordinate system is derived. For the anisotropic characteristics of the layered rock mass

the failure mode is subdivided into five kinds

for which the discrimination function

yield criterion and corresponding stress modified iteration calculation are respectively established according to plastic flow rule. A second development is carried out with VC++ programming language to generate a dynamic link library DLL file called by FLAC3D main program at any time. The strength and deformation properties of layered rock mass are studied through uniaxial and triaxial compression numerical experiments which indicate that the inclination angle and the confining pressure have an important influence on mechanical behaviors of layered rock mass. Compared with transversely isotropic elastic model as well as ubiquitous-joint model in FLAC3D

the correctness and accuracy of the new model are verified. Finally

the new model is applied to the calculation of an ideal circular tunnel in triaxial equal pressure in which the variation rule of failure modes of the layered rock mass influenced by the azimuth is analyzed. The results indicate that the altitude of bedding planes of layered rock mass is playing a control role to deformation distribution

failure mode and the propagation direction of plastic zones.