Aiming at the problems such as low flexibility, long development cycle, and incapability of adapting to rapidly changing algorithms for current video decoder implementation solutions, a generic video decoder architecture design scheme for multiple video coding and decoding standards is proposed using software and hardware collaboration. The methodology of the design is based on a programmable homogeneous multi-core processor and coprocessor hardware architecture. The homogeneous multi-core processor uses instruction-level and task-level parallel acceleration. The coprocessor uses a hardware customization unit to achieve the vector acceleration, while it uses distributed on-chip scratchpad memory instead of data cache to achieve an efficient data storage system. Taking the H.264 video standard widely used as an example, the experimental results show that the H.264 video decoder based on the architecture proposed in this paper is highly efficient and feasible, with an average speed-up of 9.12, which is 1.31 times better than the traditional multi-core parallel decoding algorithm.