In order to improve the accuracy and timeliness of fire detection and achieve the optimal utilization of existing hardware equipments, an automatic fire detection method with day-night infrared camera was proposed. Firstly, a video category classification algorithm based on IR frame color distribution in RGB color space was proposed. It can classify different videos and switch the detection state automatically. Then, fire candidate regions were extracted by using the fire color models of infrared state and visible state. Thirdly, the irregularity, corner number, frequency and correlation coefficient oscillation times were computed as four static or dynamic fire features. Finally, the training feature data optimized by Subtractive Clustering and Fuzzy C Means were the input of Neural Network,and fire classifiers for two kinds of video states were trained separately. Experiment results have indicated that the average classification accuracy of video categories has achieved 93.07%, and the proposed method is able to detect all of the 21 videos correctly. The proposed method gives warning 8 seconds after fire occurs. The processing speed is more than 25fps. It can meet the requirements of high detection precision, strong anti-interference, real-time processing speed, and adaptation to all-weather monitoring.