If you are working on ROS and want to implement simple Augmented reality applications using OpenGL and OpenCV, this post is for you. The AR applications may have the camera image on the background and drawing a 3D object on the top of the image. In this post, you can see how to draw a 3D object, say a 3D cube on the top of an image. You can apply your own algorithms to move the 3D object based on some object detection algorithms in OpenCV.
Integrating OpenGL in ROS
If you have ROS desktop full installation in your PC, you may have OpenCV installed along with the ROS installation. The OpenCV version comes along with ROS is not compiled with WITH_OPENGL=ON configuration. If it is not compiled with OpenGL, we can’t draw any 3D objects on the top of an OpenCV image.
In order to enable OpenGL on OpenCV which is installed along with ROS, you can use the following method.
Compiling Customized OpenCV for ROS
In order to enable OpenGL in OpenCV, you have to build OpenCV from source compilation.
Note: The OpenCV version of the source code should be the same as the installed opencv version. Otherwise, the OpenCV-ROS cv bridge will not work properly. In order to avoid ROS-OpenCV compatibility issues, we have to go for the same version of OpenCV. You can use the following script in order to do this task. You can either build opencv locally or you can install it on the system. The safest method is, installing locally and provide the path of opencv in CMakeLists.txt in order to build it from the local version.
You can clone the following
OpenGL_ROS package to your catkin workspace in order to get started. The package is also having the shell script to install OpenCV.
cd /<path to your catkin_ws>/src/ git clone https://github.com/qboticslabs/opengl_ros cd opengl_ros gedit scripts/install_opencv.sh
Here is the
install_opencv.sh script for installing customized OpenCV for ROS.
If you want to install customized OpenCV, first edit the configuration inside this script and save it. The main configuration parameters are
- OPENCV_INSTALL: If it ‘YES‘, it will build opencv and install into the system path. If ‘NO‘, it will build locally.
Note: I will refer you to stick with the local build of OpenCV first because it is safe to work with. Installing OpenCV may cause an undesired effect on your system.
- WITH_CUDA: If you have NVIDIA GPU and if CUDA installed on your system, you can set the config as ‘ON’. This can accelerate the processing using GPU.
- WITH_OPENGL: You have to put this configuration ON for compiling OpenCV with OpenGL.
To install the script you can simply follow the command below. Switch to
opengl_ros folder and execute the following command
It will take from 20 min to 2 hr to build OpenCV. The time will depend on your configuration. Make sure the build is successful before going to the next section. This script will create a folder called
opencv_dev_install in your home folder and clone the opencv source code to this folder. If you choose the install option, it may be installed in your system path.
After building OpenCV, you may have to do some editing in
CMakeLists.txt of the
opengl_ros package, that you have already cloned in the src folder of your catkin workspace.
I have tested this package in ROS Kinetic and Melodic. The current version of
CMakeLists.txt is for ROS Melodic. You can do some simple customization in order to make it compile with Kinetic.
Here are main sections in the
Note: For ROS Kinect users, just search for melodic in the
catkin_LIBRARIES variable and replace it with kinetic.
You can save the file and build the package in the workspace using the
If your build is successful, you can able to run the
Testing OpenGL_ROS package
In order to test the
opengl_ros node, you may use an existing package in ROS. This package will basically open your webcam and publish the images in ROS Topic: ‘
/usb_cam/image_raw‘. We can install this package using the following command
usb_cam package in ROS Melodic
sudo apt install ros-melodic-usb-cam
After installing the
usb_cam package, you can try the following commands to start the
Note: Make sure that you have webcam that is compatible with Ubuntu. Use cheese application in Ubuntu to check the webcam is working.
usb_cam node to start the camera.
roslaunch usb_cam usb_cam-test.launch
Start the OpenGL-ROS node using the following command
rosrun opengl_ros opengl_ros
Here is the output video
You can find the OpenGL_ROS code
Here is the flow of the
Acquiring ROS Image messages
If you check the
cv_gl_ros.cpp, you can find a class
ImageROS, which helps to receive and publish ROS images.
Initializing track bar and OpenGL rendering Window
You can see the initialization of the OpenGL render window and two trackbars. One is for changing the
x_rotation and others for changing
y_rotation of the 3D cube.
Render 2D texture and 3D object
on_opengl(void* param) is a callback function that is responsible for drawing the 2D texture in the background and drawing a 3D cube on top of it. In this code, the 2D texture is the input image itself. The callback read the trackbar position and render the cube orientation based on that value.
Here is the
while() loop which is doing the image acquisition, 2D and 3D rendering in OpenGL window, and checking for ‘Esc‘ key in order to quit the program.
You can find the existing integration of OpenCV-OpenGL APIs from the following link.
In this post, you have seen how to create simple Augmented reality using OpenGL, OpenCV, and ROS. We have seen how to enable OpenGL on OpenCV which is installed along with ROS. We have seen a script to install OpenCV by source compilation and have discussed how to customize in order to compile with custom OpenCV, ROS Melodic, and Kinetic. After that, we have tested the opengl_ros node and finally, you have seen a brief explanation of the code.
I’m curious why you used ROS here? It seems that it’s used only to acquire images, but the same can be done with OpenCV.
Hi @Paul. Yes, what you have said is true. If we are working on ROS ecosystem and if we want to implement an AR application on ROS , then only this will be useful. We can implement AR applications in different ways, this is when we plan to implement in ROS environment.
I am using Ubuntu 16.04 and kinetic . I got errors below when I build the package in the workspace using the catkin_make command. Do you have suggestion for this issue ? Thanks
Running command: “make -j1 -l1” in “/home/robot001/catkin_ws/build”
[ 50%] Building CXX object opengl_ros/CMakeFiles/opengl_ros.dir/src/cv_gl_ros.cpp.o
/home/robot001/catkin_ws/src/opengl_ros/src/cv_gl_ros.cpp: In function ‘void help(char)’:
/home/robot001/catkin_ws/src/opengl_ros/src/cv_gl_ros.cpp:169:6: warning: unknown escape sequence: ‘\/’
<< “\n// the OpenGL libraries cannot be found.\n\/”
/home/robot001/catkin_ws/src/opengl_ros/src/cv_gl_ros.cpp: In function ‘int main(int, char)’:
/home/robot001/catkin_ws/src/opengl_ros/src/cv_gl_ros.cpp:200:36: error: ‘CV_WINDOW_OPENGL’ was not declared in this scope
cv::namedWindow( “OpenGL_ROS”, CV_WINDOW_OPENGL );
opengl_ros/CMakeFiles/opengl_ros.dir/build.make:62: recipe for target ‘opengl_ros/CMakeFiles/opengl_ros.dir/src/cv_gl_ros.cpp.o’ failed
make: *** [opengl_ros/CMakeFiles/opengl_ros.dir/src/cv_gl_ros.cpp.o] Error 1
CMakeFiles/Makefile2:437: recipe for target ‘opengl_ros/CMakeFiles/opengl_ros.dir/all’ failed
make: *** [opengl_ros/CMakeFiles/opengl_ros.dir/all] Error 2
Makefile:138: recipe for target ‘all’ failed
make: *** [all] Error 2
Invoking “make -j1 -l1” failed
Can you use decimal value 4096 instead of CV_WINDOW_OPENGL flag? If it is not working try 0 or 1 instead
You can add an header file in the file. /catkin_ws/src/opengl_ros/src/cv_gl_ros.cpp as below.
It may fix your issue as workaround.
IN addition to above, OpenCV in kinetic uses 3.3.1-dev without OpenGL enabled. You can build from source to support OpenGL with OpenCV.
Since 3.3.1-dev version is not available from git anymore, You can replace opencv version with 3.4 in install script. and install opetion to ‘YES’,
and then need to modify “find package()( to find package (OpenCV 3.4) in your CMakeList.txt file.
This would fix your problem.