A new method of image character fuse was used to analyze and mine a mass of information of image. The mapping between the image character and its meaning was established by using neural network learning to mine the meaning of image. The image of cultural heritage was taken as the object of this study, and the images of 100 ancient buildings were selected. Their image characters and semantic information were extracted, and learning base of neural network was established. The 6 test images were selected and validated the validity and practicability of this method.