According to the structural features of power system stabilizer PSS4B, a hybrid PSO algorithm is applied to PSS4B parameter optimization in this paper. By connecting RTDS analog input/output interface card with NES6100 excitation regulator, the semi-physical simulation system is constituted, and the single machine infinite model is established on RTDS real-time digital simulation platform. Based on testing the phase frequency characteristic without compensation of generator excitation system and after parameter optimization, PSS4B critical gain experiment, step response test under load, short circuit disturbance test and inverse-regulation effect test are carried out in turn. The test results show that the optimized PSS4B can suppress power oscillation well in the different frequency bands, and have good inverse-regulation suppression effect.