Nano-CaCO3 was adopted into cement-based materials in the ultrasonic dispersion method. The influence of nano-CaCO3 with different contents on the properties and structures of cement-based materials was studied, and its mechanism was investigated by X-ray diffraction and scanning electron microscope. The results show that the fluidity is decreased, the apparent density of cement paste is increased, and the flexural strength and compressive strength are improved due to the addition of nano-CaCO3. The mechanical properties of cement-based materials containing nano-CaCO3 of 1.5% (in mass) are significantly improved. However, excessive dosage of nano-CaCO3 goes against the improvement of strength. The hydration can be accelerated, the hydration products such as Ca (OH)2 increase with the addition of nano-CaCO3 at the early age; the interface transition zone between aggregate and cement paste and the microstructure of hardened cement paste are greatly improved so that the whole structure can get more dense and homogeneous. The above results indicate that the incorporation of nano-CaCO3 is good for the physical properties and structures of cement-based materials.