Spatial pose estimation of apples for robotic harvesting

In the apple picking process, the existing robots are unable to accurately estimate the pose of apples. This often leads to damage to the fruits, branches, and even the robot fingers during grasping, while also significantly reducing the efficiency and success rate of apple picking. Hence this paper proposes a real-time 3D attitude angle estimation algorithm for apples in orchards and has independently developed a set of apple posture angle verification system. Finally, through field trials in the orchard, the effectiveness and accuracy of the algorithm are verified. Firstly, based on the fruit segmentation mask map of YOLOv8 combined with the moment of inertia algorithm, the pose angle ( 0 , θ ) and its maximum transverse diameter of the invisible calyx apple are estimated. When the model recognizes the apple calyx, the position of the core is estimated based on the surface center coordinates and the maximum transverse diameter of the apple, and then the calyx-core vector is constructed. Finally, through spatial geometry, the vectors are decomposed into the X ' ' O ' ' Y ' ' plane and the Y ' ' O ' ' Z ' ' plane respectively, and then the attitude angles ( α , β ) are calculated. The field test results of the orchard show that the accuracy rates of this algorithm in detecting apples and calyxes are 0.953 and 0.703 respectively, and the accuracy rate of fruit segmentation is 0.953. Through the calibration error test experiment, the spatial positioning error of the robotic arm is 12.1 mm. After multi-pose and multi-distance tests, the absolute error of the algorithm in estimating the maximum transverse diameter of the apple is 3.9 mm. Field experiments in orchards were conducted through the self-developed attitude angle verification system. The average absolute error of the attitude angle of apples without identified calyxes was 16.3°. Among the apple pose angles that can identify the calyx, the average absolute error is 17.0° and the average absolute error is 18.1°. The Apple pose estimation algorithm proposed in this paper provides algorithmic support for the subsequent pose capture of apple picking robots.