Unity for Control UR5 Robot

Using Oculus quest 2 to control model in Unity and send command to control actual robot

Software

• Unity
• Blender
• Oculus (Software)
• ZED SDK (Stereo Camera Software)

Blender

• Adding joint
• Adding material to the 3d model
• Convert file from STL (from SolidWorks) to FBX for using into Unity

Unity

• Create sample scene with UR model and UI for inventory
• Connect Webcam camera to Unity

• Put the real model of the robot into Unity

• Connect Oculus quest 2 controller and show in the scene
• On/Off and Switch Camera through UI

• Script for control camera

• Sending command through MQTT
• UI for choosing the bottle through button (Actual camera will streaming from stereo camera on right arm of robot

• RGB Button: Order the robot to go to the position near the bottle on the tray
• Color Bar: Show which color is on the tray
• Right-Hand Button: Use for change the UI from tray to shelf and the camera will swap from right arm to head of the robot

• Display the camera from using Sprout

Video Demo

Camera position

• Tray
• Between
• Shelf

Network

• ~1.8 – 2.5 Mb per second

Control Panel

• FW = Robot Move Forward
• BW = Robot Move Backward
• Right = Robot Move Right
• Left = Robot Move Left
• UP = Robot Move Up
• DOWN = Robot Move Down
• PICK UP = Robot No1 Pick the Bottle from Tray
• UNPICK = Robot No2 Pick Bottle from No1 and Move to Place on Shelf

Video Final

Future work

• Make it more faster and clearing

Progress Planning