A contact model considering rolling resistance is introduced and implemented into the distinct element method (DEM) for analyzing the earth pressure at critical state against a rigid wall. The variation of earth pressure coefficient and the shear band formation in the backfill material are analyzed

when the passive earth pressure varies with displacement for the retaining wall with different densities and different displacement modes (passive T mode

RB mode

RT mode). The DEM simulation results are compared with those of other researches. It is shown that earth pressure coefficient is greatly affected by different void ratios and wall displacement modes. The -displacement relation switches from hardening to softening when the void ratio decreases or the relative density increases. In contrast

the -displacement relation demonstrates either hardening or softening feature for the medium dense backfills even they exhibit a strain softening response in the biaxial compression tests. As the retaining wall moves to soil behind the wall

the shear strain increases and the strain localization occurs

i.e. shear band forms in the soil behind the wall. Similar to the shear strain field from laboratory experiment

the distributions of the shear band in the critical state are better simulated by DEM. Meanwhile

the surface of the backfill is no long a smooth surface rather than an upheaval surface

the value of upheaval increases significantly with the increase of the displacements of retaining wall

and finally the soil body behind the wall damages.