Abstract: In recent years, the threat of malware attacks on a new type power systems is increasing. In order to deal with potential risks, researchers protect the security of host systems by deploying malware detection models in smart grids. However, more and more detection models expose weaknesses in the face of carefully constructed adversarial samples. Further analysis of potential vulnerabilities is of far-reaching significance for improving the stability of a new type power systems. Therefore, this paper proposes a method to generate malware adversarial samples based on deep reinforcement learning. This method improves the evading ability of the adversarial samples by designing the attack means in the action space, and then trains the agent in a mixed environment to make the generated samples have better migration ability. The experimental results show that the adversarial sample generation method based on deep reinforcement learning D3QN has better comprehensive performance than other methods, which is conducive to further mining the vulnerabilities of current power system malware detection models.