Balloon Volleyball with a 7-DOF Robot Arm
Completed: 8 December, 2022
Teammates: Rintaroh Shima, Muye Jia, Hanbing Wu, and Ayush Gaggar
In this project, we decided to replicate the game 'Don't let the balloon touch the floor' with a Franka Emika Panda robot arm. With the help of a RealSense D435i camera, our robot is able to predict the landing location of a balloon, and move there before it hits the ground. The robot then taps the balloon back up into the air, and waits for the balloon to start falling before intercepting it again. A user may interact with the balloon to help assist in the process.
Computer Vision
My primary role on the team was to design and implement a perception pipeline that can track the balloon in a 3D space in real time. This entailed finding the centroid of the balloon and sending this over to the prediction and trajectory execution nodes. Here is an example of the final implementation of the balloon tracking.
The perception pipeline utilizes a RealSense D435i camera, which provides RGB-D data. To preserve speed, I decided to use Background Subtraction, which isolates the moving objects within the scene. This works perfect for tracking the balloon, but only if we could guarantee that the balloon is the only moving object in the scene. Unfortunately, this alone would not work since the camera would pick up random artifacts that it detected were moving. This was especially important since our prediction relied heavily on very accurate data. Thus we had to make some improvements!
In order to improve upon this, I decided to isolate only the red objects in the scene. After creating a frame that isolated the red objects, I applied background subtraction to it. When an object that fit the criteria was found, it would create a circle and find its center. Finally, it would calculate the real world (x,y,z) location of the center, and send it to the prediction and trajectory nodes.