The initial attempts to use the Kinect to capture 3d video gave developers a boost in morale and confidence. Even though there have been holes in 3d video capture (such as unreachable sides and black shades), evidence that it can be done was enough to push the ingenious minds of the community to further develop Kinect’s 3d potential. And because of that, another breakthrough in Kinect’s ability to capture 3d images has been made. Will Gady sent us this video with in-depth look at the holes in the initial phase of the Kinect’s 3D video and also the occlusion inpainting they did to correct it. In this release, he showcased the “corrected depthmap” of the Kinect and how these changes drastically improved the Kinect’s 3d prowess.
Here is a detailed description of the changes:
“One problem with using any of these is that Kinect does not provide what is technically called a “dense depth map” — there are lots of holes (black) where the Kinect IR sensor can’t “see”, whether due to lighting, objects being out of range, reflection, transparency, occlusion, and objects absorbing and not reflecting Infrared (which is how Kinect sees depth). It also won’t work too well in broad-daylight, but it was never intended to.
Once you have a dense, high-quality depthmap, you can render a high quality 3D video from it. You also need to be able to fill in disocclusions (a process known in the 3D film industry as “inpainting”), which we also do well, in real-time.
With a little of our magic fairy dust, the Kinect could become a truly workable real-time high-quality 3D camera, if only the Kinect RGB camera was higher quality. One interesting thing about the Kinect is that the RGB camera does not match the IR camera, so the depthmap has to be rectified to the RGB image. Even more interesting, our automatic rectification doesn’t care about the resolution or how far away the RGB camera is from the IR — in fact, the capture could come from different cameras. Like, let’s say hypothecially a 2K, or even 4K camera.”
For more information about Will Gady’s project, visit the Occlusion inpainting for Kinect 3D website.
Users need anaglyph glasses to view the video correctly.
