This study aims to comprehensively improve the mechanical properties, permeability and durability of the pervious concrete with recycled aggregate through optimization of cementitious matrix and adding recycled fine aggregate with different content. Firstly, the simplex-centroid design method was used to optimize the ternary cementing system composed of cement (C), fly ash (FA) and silica fume (SF), and the high-performance cementitious matrix is obtained. Then, the effects of high-performance cementitious matrix and recycled aggregate content (0%, 30% and 50%) on mechanical properties and durability of pervious concrete were analyzed. The experimental results show that the compressive strength and freeze-thaw durability of pervious concrete with recycled aggregate can be significantly improved by using high-performance cementitious materials, and it can also meet the permeability requirements. When the recycled aggregate content is 0%, 30% and 50%, the 28 d compressive strength is increased by 72.4%, 100% and 44.2%, respectively; the mass loss of 50 freeze-thaw cycles was 1.5%, 2.2% and 2.5%, respectively. In addition, it is found that the failure mode of pervious concrete with recycled aggregate is related to the performance of cementitious material and the content of recycled aggregate, which can provide a reference for the design and application of pervious concrete with recycled aggregate.