For the large open-pit mining

there is still a large amount of residual coal in highwall mines. However

the reasonable width of rib pillar is always a difficult problem

which has greatly restricted the high-efficiency application of highwall mining system. The working faces in highwall mining were assumed as a series of co-linear cracks with certain spacing interval. Based on theⅠ-Ⅱ composite crack model in fracture mechanics

the stress distribution at the edge of rib pillar was analyzed

combined with Hoek-Brown failure criterion and Mohr-Coulomb failure criterion. Besides

the boundary equations of damaged zone at the edge of rib pillar were established. Accordingly

the calculation formulas of damaged width of rib pillar were obtained under different failure criteria

respectively. The curve laws were presented on the basis of the theoretical formulas of damaged width of rib pillar under different failure criteria and three sets of characteristic parameters for determining different states of coal body damage. According to the curve laws under different failure criteria

the effects of independent factors

such as the width of retaining coal pillar

the width of cavern

burial depth

H

and the dip angle of coal seam

?

on the damaged width of rib pillar

were analyzed. Meanwhile

the data of field tests were substituted into the theoretical formulas under different criteria. The theoretical results and measured data showed that the mutative laws obtained by applying the Hoek-Brown failure criterion accorded with the evolutional laws of rib pillar damage of actual field better

and the application of Hoek-Brown failure criterion was more suitable for the analysis of fracture of jointed rock mass. In addition

the mechanism and process of rib pillar destabilization were described by considering the limit failure theory and the asymptotic failure theory in the stability study of coal pillar. This paper provides a new approach to calculate the reasonable width of rib pillar and the application of criteria in theoretical analysis of pillar failure

which is significant for popularizing the highwall mining technology.