Sunday, 29 December 2013

OpenCVKinect: Acquiring Kinect Data Streams in OpenCV

Click here to go to code download step directly
_______________________________________________________________________________________________

Edit (26/05/2016) : I have updated the OpenCVKinect to fix some bugs and make use of different visualization for depth maps. Details of this newer version can be seen here. All other details mentioned in this blog post still apply to the updated version.
_______________________________________________________________________________________________

Holiday season is here, and I have finally found sometime to write something for this blog. Sometime back I wrote a guide for compiling OpenCV with OpenNI support. Since that post a lot has been changed in the newer OpenNI 2.x SDK, there has been a lot of improvements for using Microsoft Kinect Sensor. One of the major change is the Object Oriented Implementation as opposed to pure C implementation in previous OpenNI 1.x versions. I have had many requests asking me to write a tutorial for compiling OpenCV with the new OpenNI 2.x support. Although this is not possible with the current SDK and OpenCV, this blog post introduces an implementation for acquiring Kinect Data Streams in OpenCV.

After a long time, I recently started working with Kinect Sensor again, and therefore I wanted to access the data streams in a more efficient manner. While it looked straightforward to use the built in function calls in OpenNI 2.x, I was more comfortable with using OpenCV format instead. Therefore I wanted to hide all that detail of OpenNI Objects and their methods in a simple and convenient way. To achieve this, I wrote my own implementation and this blog post officially releases this code as open source, for anyone to use/modify/update to. Right now I have some basic methods, but in future I am thinking of adding simple image processing/data recording methods too (to make it easier for beginners).

The code is dependent on both OpenCV and OpenNI binaries and unlike my previous post, this one does not needs compiling OpenCV with OpenNI support. Every SDK used here is used as it is, without any special compiling parameters. The only additional information is that you will be required to include library paths and link precompiled OpenCV and OpenNI binaries to your project. For completeness everything is explained below in steps, if you are aware any/most of these steps then you can skip to this step to get the code and start developing.

Setting up Libraries

The code for OpenCVKinect was written in Windows environment, therefore this post will explain everything related to using it in Windows set up. Nevertheless it can be easily used on any other platform.

To set up everything, download and install Kinect SDK v1.8 followed by OpenNI SDK v2.2 (I have tested with x86 but might also work with x64). As opposed to version 1.x of OpenNI, version 2.x does not conflict with Kinect SDK, infact some driver functionality from Kinect SDK is used in OpenNI 2.x. This is the reason for first installing Kinect SDK and then OpenNI SDK.

If you already have OpenCV compiled and installed for Microsoft Visual C++, then skip to the next part of this post. Otherwise, download the latest OpenCV precompiled binaries for Microsoft Visual C++ and follow the instructions in my previous post to set up the environment variables.

Using the Existing Code

Get/fork the code from github repository for this implementation at: https://github.com/masad801/OpenCVKinect

This repo contains an MS VC++ project with everything set up, and a working example showing the usage of the OpenCVKinect class. The project settings point to the location of OpenCV and OpenNI libraries on my hard drive, which you might have to change to match the location of these libraries on your local PC.

Creating a New Project

If existing project from Github repo does not work out for you, then you can create an Empty Console Project in MS VC++. Apply the project settings for OpenCV and OpenNI. Once the project is set-up and linked to all the libraries, add the source code from:
  • Code downloadOpenCVKinect.zip (there are three files, two for the OpenCVKinect class and one main source)
If everything is set up fine then you will be able to access and display the depth and color data streams from Kinect sensor.

Thats it for now. I hope this code helps anyone facing the problem of quickly accessing the data streams from Kinect. I will be adding more details and possibly some description/explanation of code here soon. Until then Good night and Happy Holidays!!

24 comments:

  1. Assalam,

    brother Muhammad, i'm saiful hadi from malaysia, i would like to ask a question, if i use the kinect X-box is it necessory to install the kinect SDK v1.8?

    ReplyDelete
    Replies
    1. Yes! you have to follow the instructions mentioned above. With the OpeNNI SDK 2.0 and onwards you need Kinect SDK in order to access the data streams on the Kinect Sensor.

      Delete
  2. Assalam,

    brother Muhammad, how to accessing IR video stream using OpenCV245, OpenNI2, NiTE2, Kinect SDK and VS2010C++

    Wassalam

    Saipol Hadi Hasim
    hasim_hadi@oum.edu.my

    ReplyDelete
  3. Hi, I have tried the code but I have a problem with depth stream and color stream only.
    The problem is :"Expression:vector subscript out of range" and with debug I have seen that the problem is here "dataStream[0];" because the stream is empty.

    With both camera(in third and fourth menu) the code functions without problems.

    Could you help me? Thank you!

    ReplyDelete
    Replies
    1. Hey, I will check this and get back to you. It is most probably problem with vector push_back, looks like its not returning anything when in single data stream mode.

      Delete
    2. Hey, I have solved the problem with vector push_back with:
      - a series of control in main.cpp
      " if(dataStream.size()==1 && !dataStream[0].empty())
      cv::imshow("Depth", dataStream[C_DEPTH_STREAM]);
      else if(dataStream.size()==1 && !dataStream[1].empty())
      cv::imshow("Color", dataStream[C_COLOR_STREAM]);
      else
      { cv::imshow("Depth", dataStream[C_DEPTH_STREAM]);
      cv::imshow("Color", dataStream[C_COLOR_STREAM]);
      }"

      - a different use of returnVec.push_back in OpenCvKinect.cpp.

      I share my solution thus I can help the others have the same problem of mine (https://www.dropbox.com/s/qdwuq661yvnzaw3/OpenCVKinect-master.rar).

      However I have other two problems:
      - the depth stream is extremely dark and I don't know how I can improve it.
      - sometimes the stream in general is slow and not fluid and finally the thread and stream is closed. What is the reason and how can I solve it?

      Thank you very much for your tutorial and your willingness!

      Delete
    3. Many thanks.

      This problem is related to something in the class I wrote. I am currently working with previous version of OpenNI. When I get the chance I will set everything up and update the code.

      As far as the depth image is concerned, the problem of it being dark is related to how a depth image is stored and what it represents. There is some explanation to this in a newer post: http://seevisionc.blogspot.co.uk/2014/01/reading-kinect-depth-image-in-opencv.html

      Normally images are 8bit and contain values from 0 to 255. Whereas a depth image is represented with 16 bits, which is capable of representing 65536 different values. Unfortunately this can not be easily displayed as the display methods assume an image to be 8 bit and hence only represent values which go from 0 to 255. This is the reason why it appears dark as only first 8 bits are displayed.

      To display it properly you need to scale down the values from 0 to 65536 to 0 to 255.
      check convert function in opencv Mat type for doing this.

      I hope this helps.
      Asad

      Delete
    4. I get the same error as Francesco. I hope he can help me?

      Delete
  4. Hey, thank you very much for the answer but unfortunately I have not yet solved the problem.

    I have read your recent post but I don't know how to see depth image such as, for example, in the example of OpenNI NIViewer. The depth stream is extremely dark and I don't know how I can improve it.

    I take this opportunity to ask you another simple question. How can I find depth in a specific point? I have x and y coordinates but I don't find the third coordinate.

    Thank you very much for your willingness!

    ReplyDelete
    Replies
    1. This is how you do it: http://stackoverflow.com/questions/13162839/kinect-depth-image/13201191#13201191

      I will try to include a visualization function in the OpenCVKinect library, however at the moment I do not have enough time to do this. Maybe this weekend, but I cant promise anything.

      Delete
  5. Thank you very much for your answer and your time!

    With this simple conversion "m_depthImage.convertTo(m_depthImage, CV_8U, 0.05f);" the result will change from this(http://i59.tinypic.com/2z8szew.png) to this(http://it.tinypic.com/view.php?pic=25a6a38&s=8#.U1jhjfl_vHU).

    The last question that I have is about how to search the third coordinate in a depth map.
    If you don't have time you could advise me any tutorial or any way to solve my problem? I have seen that in the sample NiViewer(https://github.com/OpenNI/OpenNI2/tree/master/Source/Tools or https://github.com/OpenNI/OpenNI/tree/master/Samples/NiViewer) of OpenNI there is this value (with the name of "pointer value" in the image http://i61.tinypic.com/2vw5oaa.png) but unfortunately I don't find the code for this information.

    If you can help me I will be extremely grateful!

    Thank you very much for your time!

    ReplyDelete
    Replies
    1. From my previous blogpost at:
      http://seevisionc.blogspot.co.uk/2014/01/reading-kinect-depth-image-in-opencv.html

      "This is, obviously, not true for a Kinect Depth Image, for this image is a special type of grayscale image. The Kinect Depth Image contains only one channel (like any other grayscale image), however the depth of this image is actually UINT16 or unsigned int (CV_16UC) instead of UCHAR. This difference in depth is because of the fact that a Kinect Depth Image contains more values and hence it requires more bits per pixel to store this information (i.e. 16bits/pixel). Now that we know what makes it different, lets see how it can be read inside OpenCV code."

      The Kinect depth image is just a special type of grayscale image. This means that it only has one channel. You have your x and y coordinates, accessing the depth is as simple as accessing a grayscale value in an image. The actual value store at the pixel corresponding your x and y coordinate is your depth at that location (i guess you might already have an idea on how to access color/grayscale pixels).

      Delete
  6. Hello,

    I tried to set up your code however I got the following crash after launch:

    The program '[6736] OpenCVKinect.exe' has exited with code -1073741701 (0xc000007b)

    This happens right after I get an error that OpenCVKinect.pdb is outdated. It asks me to delete it and rebuild the solution. I use clean solution; rebuild and I get the error...

    Any idea what it can be related to? (using VS 2013 on 64 bit windows (building on 32 bit compiler))

    ReplyDelete
    Replies
    1. Although I have no certain solution which might work, there are a number of things you can try.

      I have had trouble using OpenCV with VS2012. Might be the same issue, can be sure.
      The above project is build on VS2010. Would it be possible for you to try it on VS2010 and see if the problem persists.

      Also try making a new project in VS2013 and adding the existing code from the above link. Might work.

      Delete
  7. Hello,

    I try to run your code but a window with the following message appears: "the program can't start because openni2.dll is missing from your computer", then the program crashes.

    I am using VS2013, with Kinect SDK 1.8, OpenCV 2.4.10 and OpenNI 2.2. I think the project is set up correctly with the appropiate paths to the libraries and binaries.

    Thank you and I'd appreciate any help.

    ReplyDelete
    Replies
    1. The problem is that your application is compiling and linking properly, however on the runtime it is not able to find the libraries on which it is dependent.

      There are two solutions to this:
      Easy way- Temporary --- From OpenNI binary folder (which is normally found at: $OpenNIInstallDir$\Bin\) copy the dll file OpenNI2.dll to your Debug or Release folder containing your executable.

      Hard way-Done once for all future project --
      You will need to add the path to OpenNI Binaries (again its a Bin folder inside your OpenNI folder) to your envoirnment variables in windows. I have previously written how this is done so follow this link and look for "Setting up pc to use the compiled binaries": http://seevisionc.blogspot.co.uk/2012/07/compiling-opencv-with-openni-for.html

      Do either of these and you will be good to go!

      Delete
    2. Hello,

      I managed to find the bin folder with OpenNI2.dll, inside the OpenNI2 samples folder (weird). Anyways I added it to the Executable Directories in the properties of the VS project. And it works! kinda...

      The first two options of the program (Color and Depth stream only respectively) cause a Debug Assertion Fail "Expression: vector subscript out of range".

      The 3rd and 4th work perfectly.

      You've been really helpful. Thank you and I hope I don't bother you with this questions, I'm new at Kinect and OpenCV/NI programming.

      Delete
  8. Hello,
    Which all kinect version can be used for the above procedure ?? We are working on a project to develop tutorial for children .Please help !

    ReplyDelete
    Replies
    1. This has been tested on Kinect for Xbox 360.

      Delete
    2. Does is require any adapter ? http://www.amazon.in/Xbox-Kinect-Adapter-Power-Supply/dp/B004IXRXGY can this adapter be used if required ? or any other external device? please specify

      Delete
  9. Hi, I am a student and i am starting work wth kinect to make my final year prject i.e virtual dressing room.so can you please help me to know that how i superimpose or try-on clothes using kinect and opencv. please help me.

    ReplyDelete
  10. Would it be wrong if I use the Kinect SDK 1.5 version?

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. How do i save the kinect depth image using opencv and python in png format? ..anyone please ..i am very desperate to know

    ReplyDelete