Unity for Control UR5 Robot


  • Unity
  • Blender
  • Oculus (Software)
  • ZED SDK (Stereo Camera Software)


  • Adding joint
  • Adding material to the 3d model
  • Convert file from STL (from SolidWorks) to FBX for using into Unity


  • Create sample scene with UR model and UI for inventory
  • Connect Webcam camera to Unity
  • Put the real model of the robot into Unity
  • Connect Oculus quest 2 controller and show in the scene
  • On/Off and Switch Camera through UI
  • Script for control camera
  • Sending command through MQTT
  • UI for choosing the bottle through button (Actual camera will streaming from stereo camera on right arm of robot

  • RGB Button: Order the robot to go to the position near the bottle on the tray
  • Color Bar: Show which color is on the tray
  • Right-Hand Button: Use for change the UI from tray to shelf and the camera will swap from right arm to head of the robot
  • Display the camera from using Sprout

Video Demo

Camera position

  • Tray
  • Between
  • Shelf


  • ~1.8 – 2.5 Mb per second

Control Panel

  • FW = Robot Move Forward
  • BW = Robot Move Backward
  • Right = Robot Move Right
  • Left = Robot Move Left
  • UP = Robot Move Up
  • DOWN = Robot Move Down
  • PICK UP = Robot No1 Pick the Bottle from Tray
  • UNPICK = Robot No2 Pick Bottle from No1 and Move to Place on Shelf

Video Final

Future work

  • Make it more faster and clearing

Progress Planning

One comment

  1. Pingback: FRA631 : Foundations of Robotics – Human-Computer Interface Lab

Comments are closed.