This paper introduced a machine vision imaging system to acquire rail surface images based on line scanning, and presented an algorithm to detect rail surface defects accurately based on image enhancement and automatic thresholding. We proposed a local zero mean measure to enhance rail images, which can overcome the nonuniform reflection of the rail surface and improve the distinction between defects and background. And then, we put forward a proportion emphasizing maximum background-class variance measure to select a threshold, which maximizes the background-class variance and meanwhile keeps the defect proportion in a low level. Through experiments, we compared the core of the algorithm with well-established methods, and then proved the validity and rapidity of the algorithm with wide applicability.