To address the difficulty in extracting fault features of planetary gearboxes, this paper proposes a gearbox fault diagnosis method that uses an improved Newton-Raphson-based optimization (INRBO) algorithm to optimize the parameters of feature mode decomposition (FMD). First, a multi-strategy improvement approach is proposed to comprehensively enhance the optimization performance of NRBO. Then, the adaptive parameters of FMD are optimized using INRBO to construct a wind turbine gearbox fault diagnosis model based on INRBO-FMD. Finally, the improved method is applied to perform adaptive feature mode decomposition on the raw vibration signals of the gearbox, and fault diagnosis research is conducted on the planetary gearbox. Experimental results show that the proposed INRBO-FMD method can effectively extract the characteristic fault frequencies and harmonic components, verifying its accuracy and practicability.