Using Oculus quest 2 to control model in Unity and send command to control actual robot
Software
- Unity
- Blender
- Oculus (Software)
- ZED SDK (Stereo Camera Software)
Blender
- Adding joint
- Adding material to the 3d model
- Convert file from STL (from SolidWorks) to FBX for using into Unity
Unity
- Create sample scene with UR model and UI for inventory
- Connect Webcam camera to Unity
- Put the real model of the robot into Unity
- Connect Oculus quest 2 controller and show in the scene
- On/Off and Switch Camera through UI
- Script for control camera
- Sending command through MQTT
- UI for choosing the bottle through button (Actual camera will streaming from stereo camera on right arm of robot
- RGB Button: Order the robot to go to the position near the bottle on the tray
- Color Bar: Show which color is on the tray
- Right-Hand Button: Use for change the UI from tray to shelf and the camera will swap from right arm to head of the robot
- Display the camera from using Sprout
Video Demo
Camera position
- Tray
- Between
- Shelf
Network
- ~1.8 – 2.5 Mb per second
Control Panel
- FW = Robot Move Forward
- BW = Robot Move Backward
- Right = Robot Move Right
- Left = Robot Move Left
- UP = Robot Move Up
- DOWN = Robot Move Down
- PICK UP = Robot No1 Pick the Bottle from Tray
- UNPICK = Robot No2 Pick Bottle from No1 and Move to Place on Shelf
Video Final
Future work
- Make it more faster and clearing
Progress Planning
