Put a self-learning robot by a conveyor and let it learn for itself what to do. A repository for EC601 Product Design Section A2 Team 5 project.
Yongjie Yang - [email protected]
Sen Zhang - [email protected]
If you like this project, please give us a star
Analyze images to identify specific items, training the Object Detection method which we need in the picture, and then control the robot arm to implement the capture task.
Computer Language: Python
The conveyor delivers objects. The robot should identify the object, grab it, and move it to the destination.
- Image processing
Image processing is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known.
- Robotic arm control system
A control system manages, commands, directs, or regulates the behavior of Robotic Arm using control loops.
- Overall integration on Unity
Focusing on basic implementation of image processing and robot machine learning. Sprint 2
Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos.In our project, we only focus on specific items on the conveyor belt.
The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents. Agents can be trained using reinforcement learning, imitation learning, neuroevolution, or other machine learning methods through a simple-to-use Python API.
Build a new basic Unity game of a ArmRobot
The Python program takes screenshots continuously and stores them in a certain directory for access. The screenshot is very small (409 pixels) and only covers the key area where object identification will happen. The image will be transformed into an array (by using the package numpy) and then analyzed.
Integration our Unity game and Image Processing
For easy control of the program, we made an interface using tkinter package. By clicking on the buttons on the interface, we can choose which objects to grab, or whether the arm will grab anything at all. There are five options: Spheres, Cubes, Cuboids, All objects, and No object.
We made an introduction poster and a video demo for this project. You can take a brief look for our poster and you can also download our video for our detailed demonstration.