To address the problem of poor point cloud plane fitting effect and easy misidentification of the random sample consensus (RANSAC) algorithm for noisy point cloud data, improvements to the algorithm are necessary. The proposed algorithm employs density-based spatial clustering of applications with noise (DBSCAN) to modify the selection strategy of the initial point set in the RANSAC algorithm and uses principal component analysis (PCA) to compute the normal vectors of each point in the point cloud. Two constraint conditions, the distance from the point to the plane and the angle between the normal vectors of the point and the plane, are simultaneously used as the criteria for determining the points within the RANSAC algorithm model. The point cloud simulation data with no noise and 300 and 500 noisy points are used for testing. The fitting results of the proposed algorithm are all approximate to the theoretical values, and the standard deviations of the inner point distance are 1.007×10-8, 0.003 and 0.007, respectively, better than those of RANSAC algorithm. Using actual workpiece point cloud data for testing in two operating scenarios, the proposed algorithm improves the fitting ratio of in-plane points by 24.7% and 24.6% compared to the traditional RANSAC algorithm, respectively. The completeness and accuracy of plane extraction are also superior to those of the RANSAC algorithm. Simulation and case analysis validate the effectiveness of the proposed algorithm.