Abstract: The participation of an energy storage system in the frequency modulation has become an effective means to stabilize the frequency fluctuations of a power system. Reasonable operation mechanism of the auxiliary service market is conducive to encouraging the energy storage to provide high-quality auxiliary services and cultivating high-quality third-party market players. In view of the problems faced by the existing auxiliary service market with the participation of energy storage in the frequency modulation, such as the imperfect pricing mechanism, the unreasonable resource compensation mechanism of the fast and slow frequency modulation, and the excessive limit of the energy storage capacity, an operation mechanism of the auxiliary market of the frequency modulation is designed to adapt to the energy storage participation. Under this market mechanism, an optimal scheduling strategy of the frequency modulation resources is proposed considering the cost of frequency modulation and the recovery of energy storage SOC. Firstly, the weight coefficients of the comprehensive frequency modulation performance indexes of different types of frequency modulation resources are calculated by using the subjective and objective weight method, discribing the differences of the frequency modulation performances of multiple types of the frequency modulation resources. Secondly, the frequency modulation demands in different trading periods horizontally are compared and the time relaxation factor is introduced to optimize the marginal clearing prices during the trading period, which is beneficial to form an frequency modulation price signal that effectively maps the frequency modulation demand changes in the system. At the same time, in order to quickly clarify the difference of the frequency modulation effects of the fast and slow frequency modulation resources, the spatial relaxation factor is used to quantify the alternative capacity of different frequency modulation resources, forming a reasonable compensation for the fast and slow frequency modulation resources. Then, fully considering the influence of energy storage state, the minimum frequency modulation cost within the scheduling cycle is taken as the objective function. The real-time control optimization strategy of the multi-type frequency modulation resources is proposed, which reduces the frequency modulation cost and effectively avoids the excessive limit of the energy storage capacity. Finally, based on the current frequency modulation auxiliary service market trading rules in Guangdong province, the practicability and effectiveness of the proposed market mechanism and the scheduling strategies are verified.