3-D Detection & tracking for semi-deformable objects

Abstract

This thesis introduces a computer vision system designed for real-time detection and pose estimation of semi-deformable objects in 3-D space, leveraging edge computing devices. The primary motivation for this research stems from the need to enhance the capabilities of vision-based systems, which in turn can aid in improving the efficiency and effectiveness of robotics systems in a variety of fields. For the context of the thesis the chosen field was agriculture, focusing on the recognition, tracking and pose estimation of bell peppers by harvesting robots, an application where traditional methods often fall short due to the nature of semi-deformable objects like fruits. A Jetson Nano was used as the main component, while an Intel DE10-Nano was considered as a complementary part of the system for performing image preprocessing tasks with the Azure Kinect being considered as the main camera sensor. The algorithm was successfully deployed in the Jetson Nano, successfully tracking and estimating the pose of a bell pepper in 3-D by performing the necessary rotations and deformations to a canonical model used by the system as a general means to estimate the pose of the pepper in the real world scene. The algorithm was also tested in a ROS 2 Gazebo simulation where an x-arm robot was used to simulate the vision part of a pick and place operation with a simulated bell pepper, using the proposed method to accurately identify and estimate the pose of the pepper in the simulation. Lastly, a set of different segmentation techniques using both deep learning and traditional methods are presented as a means to explore how these could better the current segmentation capacity of the system.