Derivative is not defined at the edge corners of yield and plastic potential surfaces based on the generalized Hoek-Brown strength criterion and numerical singularity has been caused there. Smooth transitions with circular surface only satisfy the first derivative continuity; the second derivative is still undefined on the edges; so that the consistent tangent modulus can not be calculated correctly there. The implicit solution of nonlinear equilibrium equations in finite element method(FEM) with Newton-Raphson method is unable to achieve full quadratic convergence. C2 continuous rounding of yield and plastic potential surfaces in the π-plane has been introduced; so that the surfaces can be second-order continuous derivative on the edges; and the consistent tangent modulus matrix can be calculated accurately. Subroutine UMAT for the generalized Hoek-Brown linear elastic-perfectly plastic constitutive model has been coded with FORTRAN in ABAQUS software. Numerical examples are used to verify the correctness of the proposed method.