Abstract: The widespread integration of distributed renewable energy sources has brought a series of problems to the operation of distribution networks, including voltage violations and increase in network losses. This paper proposes a model-free voltage control strategy based on multi-agent reinforcement learning. By coordinating photovoltaic inverters, distributed energy storages, and soft open points, the strategy aims to reduce network losses and eliminate voltage violations. To tackle the problem that traditional voltage control strategies have strong dependence on accurate distribution network model parameters, a power flow surrogate model based on Gaussian process regression is proposed. The model enables offline training and online application through interactions between multi-agents and the power flow surrogate model. Additionally, a multi-agent deep reinforcement learning algorithm based on random weighted triple Q-learning is proposed to further reduce the overestimation and underestimation errors of the soft actor-critic algorithm. The proposed method improves the algorithm exploration capability and results quality. Finally, simulation results on the IEEE 33-node system verify the effectiveness of the proposed method in solving the distributed voltage control problem of distribution networks.