In order to solve the problem of vehicle stability control after single wheel brake failure in the brake-by-wire system, the control strategy of optimal braking force distribution was proposed by cooperating the steer-by-wire and brake-by-wire systems. To meet the driver's desired deceleration as closely as possible, the braking force of the remaining three wheels was redistributed initially by the sequential quadratic programming method. The front steering controller was designed by sliding mode control method in order to prevent the vehicle from deflection and spin caused by the braking force reconstruction. Considering the influence of the front steer on the longitudinal force of the tire, the mathematical model of tire lateral force based on the magic formula was established. The braking force was optimized in the real time under the side slippery condition by the sequential quadratic programming method. A co-simulation experiment established by Simulink and Carsim was carried out to analyze the strategy, and the result showed that the yaw rate of the vehicle quickly converged to 0 and the lateral deviation displacement was less than 0.1 m. The result verifies that the proposed method of optimal braking force distribution can improve the braking stability of single wheel brake failure vehicle under different braking conditions.