3D‐vision based detection, localization, and sizing of broccoli heads in the field

AbstractThis paper describes a 3D vision system for robotic harvesting of broccoli using low‐cost RGB‐D sensors, which was developed and evaluated using sensory data collected under real‐world field conditions in both the UK and Spain. The presented method addresses the tasks of detecting mature broccoli heads in the field and providing their 3D locations relative to the vehicle. The paper evaluates different 3D features, machine learning, and temporal filtering methods for detection of broccoli heads. Our experiments show that a combination of Viewpoint Feature Histograms, Support Vector Machine classifier, and a temporal filter to track the detected heads results in a system that detects broccoli heads with high precision. We also show that the temporal filtering can be used to generate a 3D map of the broccoli head positions in the field. Additionally, we present methods for automatically estimating the size of the broccoli heads, to determine when a head is ready for harvest. All of the methods were evaluated using ground‐truth data from both the UK and Spain, which we also make available to the research community for subsequent algorithm development and result comparison. Cross‐validation of the system trained on the UK dataset on the Spanish dataset, and vice versa, indicated good generalization capabilities of the system, confirming the strong potential of low‐cost 3D imaging for commercial broccoli harvesting.