Mesh vertex spacing circa 1cm.

[wptabtitle] Cloud Compare[/wptabtitle] [wptabcontent]

The open source software Cloud Compare, developed by Daniel Girardeau-Montaut, can be used to perform this kind of assessment. The assessment method described here is based on the application of a series of perceptual metrics to 3D models. In this example we compare two 3D models of the same object, one derived from a C10 scanner and one from from a photogrammetric model developed using Agisoft Photoscan.[/wptabcontent]

[wptabtitle] Selecting Test Features[/wptabtitle] [wptabcontent]

Shallow but broad cuttings decorating stones are common features of interest in archaeology. The features here are on the centimetric scale across (in the xy-plane) and on the millimetric scale in depth (z-plane). In this example we assess the resolution at which a characteristic spiral and circles pattern, in this case from the ‘calendar stone’ at Knowth, Ireland is legible, as recorded by a C10 scanner at a nominal 0.5cm point spacing, and by a photogrammetric model built using Agisoft’s photoscan from 16 images.

[/wptabcontent]

[wptabtitle] Perceptual and Saliency Metrics[/wptabtitle] [wptabcontent]

Models from scanning data of photogrammetry can be both large and complex. Even as models grow in size and complexity, people studying them continue to mentally, subconsciously simplify the model by identifying and extracting the important features.

There are a number of measurements of saliency, or visual attractiveness, of a region of a mesh. These metrics generally incorporate both geometric factors and models of low-level human visual attention.

Local roughness mapped on a subsection of the calendar stone at Knowth.

Roughness is a good example of a relatively simple metric which is an important indicator for mesh saliency. Rough areas are often areas with detail, and areas of concentrated high roughness values are often important areas of the mesh in terms of the recognizability of the essential characteristic features. In the image above you can see roughness values mapped onto the decorative carving, with higher roughness values following the edges of carved areas.

[/wptabcontent]

[wptabtitle] Distribution of Roughness Values[/wptabtitle] [wptabcontent]The presence of roughness isn’t enough. The spatial distribution, or the spatial autocorrelation of the values, is also very important. Randomly distributed small areas with high roughness values usually indicate noise in the mesh. Concentrated, or spatially autocorrelated, areas of high and low roughness in a mesh can indicate a clean model with areas of greater detail.

High roughness values combined with low spatial autocorrelation of these values indicates noise in the model.

[wptabtitle] Picking Relevant Kernel Sizes[/wptabtitle] [wptabcontent]

To use the local roughness values and their distribution to understand the scale at which features are recognizable, we run the metric over our mesh at different, relevant, kernel sizes. In this example, the data in the C10 was recorded at a nominal resolution of 5mm. We run the metric with the kernel at 7mm, 5mm, and 3mm.

Local roughness value calculated at kernel size: 7mm.

Local roughness value calculated at kernel size: 5mm.

Local roughness value calculated at kernel size: 3mm.

Visually we can see that the distribution of roughness values becomes more random as we move past the effective resolution of the C10 data: 5mm. At 7mm the feature of interest -the characteristic spiral- is clearly visible. At 5mm it is still recognizable, but a little noisy. At 3mm, the picture is dominated by instrument noise.
[/wptabcontent]
[/wptabs]

[wptabtitle] Metadata and Headers.[/wptabtitle] [wptabcontent]

Header information read by LASTools

Essential information about the data itself, organization and initial processing of a LAS file is contained in its header. Lidar processing software including LASTools and LP360 will allow you to access the LAS header information. It’s always a good idea to look at the headers to learn things like:

• The software used to generate the file
• The number of returns
• The total number of points
• Offsets and scale factors applied

Header information read by LP360

Many data providers will also supply a detailed project report including information on the project’s error budget, ground control networks, flight conditions, and other technical details. [/wptabcontent]

[wptabtitle] Checking the Point Density.[/wptabtitle] [wptabcontent]Knowing the real resolution of your lidar data is important. Checking that it matches your requested resolution is an essential part of quality control in an ALS project. This information will affect the parameters you select for classificaton and interpolation; It may also influence your expectations regarding the types of features you should be able to identify or accurately measure.

In LASTools you can use the ‘-cd’ or ‘-compute_density’ option in LASInfo to compute a good approximation of the point density for the file. Alternatively, you can use SAGA GIS, an open source GIS software package, to plot per grid cell density and visualize how the densities vary across your dataset.

Points per grid cell visualized in SAGA.

Histogram of points per grid cell, visualized in SAGA.

[wptabtitle]Sources of ALS Errors.[/wptabtitle] [wptabcontent]The total error for a lidar system is the sum of the errors from the laser rangefinder, the GPS and the IMU. These sources of error and the calculation of error budgets have been discussed extensively in the literature, including good summaries by Baltsavias (1999) and Habib et al. (2008). For ALS surveys conducted from fixed-wing aircraft platforms, these often total somewhere between 20 and 30cm.

The main sources of error are:

• Platform navigational errors
• GPS/IMU navigational errors
• Laser sensor calibration errors (range measurement and scan angle)
• Timing resolution
• Boresight misalignment
• Terrain and near-terrain object characteristics

Errors may be vertical (along the Z axis) or planimetric (shifts on the XY plane). The errors are obviously related, but they are usually quantified separately in accuracy reports. In commercial applications accuracy analyses usually focus on vertical accuracy, while planimetric accuracy (XY) is secondary.
[/wptabcontent]
[wptabtitle]Types of Errors.[/wptabtitle] [wptabcontent]Both horizontal and vertical errors may be described as random, systematic or terrain dependent. The main source of random error is position noise from the GPS/IMU system, which will produce noise in the final point cloud. These coordinate errors are independent of the flying height, scan angle and terrain.

Systematic errors include errors in range measurement, boresight misalignment, lever arm offset and mirror angle, and some errors from the GPS/IMU system (e.g. INS initialization and misalignment errors and multi-path returns). These errors will appear throughout the dataset. Terrain dependent errors derive from the interaction of the laser pulse with the ojects it strikes. In steeply sloping terrain or areas with off-terrain objects, and at higher scan angles, beam divergence may be increased and result in vertical errors due to horizontal positional shift.

Errors are most visually apparent in areas of strip overlap. A characteristic sawtooth pattern seen in hillshaded DTMs and clear misalignments of planar roof patches seen in the profile are typical of misalignment between adjacent strips.

Strip overlap errors seen in a hillshaded DTM.

Two scans of the same roofline in two overlapping strips are slightly offset, indicating a slight error. Points coloured by flightstrip.

[wptabtitle] Classification Errors.[/wptabtitle] [wptabcontent]Lidar data is typically gathered across large areas of the landscape, which may include woodland, urban and arable areas. One of the advantages of lidar over other remote sensing technologies is its ability to ‘see through’ the vegetation canopy, as some returns will pass through gaps in the canopy, reaching and returning from the ground- allowing the creation of a bare earth DEM. To accomplish this data must be classified (or filtered) to separate returns from terrain and off-terrain objects.

Points incorrectly classified as low vegetation (dark green) which should be terrain (orange).

There are a number of algorithms in use for classifying a point cloud. Regardless of the algorithm used, some errors will be committed. Two types of classification errors occur when performing a classification: the removal of points that should be retained (type 1) and the inclusion of points that should be removed (type 2). Overly aggressive algorithms or parameter settings have a tendency to remove small peaks and ridges in the terrain and to smooth or flatten the ground surface. Conversely, insufficiently aggressive parameters will induce the inclusion of clumps of low vegetation returns in the ground class, and can result in false ‘features’ [/wptabcontent] [/wptabs]

Hydro-enforced DTMs include breaklines, importantly stream centerlines and edges, and breaklines delimiting standing water bodies such as ponds. While auto-extraction of breaklines is improving, the creation of hydro-enforcing features is still by and large a manual task.

Deliverables include hydro-DTMs.

[wptabtitle] DSMs[/wptabtitle] [wptabcontent]Digital Surface Models (DSMs) can include only returns from the terrain, buildings and specific classes of off-terrain objects like bridges, or can also incorporate returns from vegetation. DSMs are commonly used in urban environment analyses such as noise pollution modeling and inter-visibility analyses to assess the impact of new building.

DSMs are often used for modeling in urban areas.

[wptabtitle] CHM[/wptabtitle] [wptabcontent]Canopy height models, and per-stand or individual tree metrics are important ALS-based products for forestry applications. These models often include returns separated into low- mid- and high- vegetation classes, and are sometimes normalized based on local terrain heights to facilitate comparisons between different forest areas.

Canopy height model generated using SAGA GIS.

[wptabtitle] Contours[/wptabtitle] [wptabcontent]Contour maps at standard intervals, e.g. 1m, 5m, or 20m contours, can be generated from bare earth DTMs. Contour maps can be generated with or without breaklines.

Contours developed based on the terrain model.

A tie strip can be seen here overlapping with two flightlines.

[wptabtitle] Tile Schemes[/wptabtitle] [wptabcontent]Because ALS datasets are usually very large, they are often divided into regularly sized tiles. These tiling schemes can help with the speed of data loading, and allow users to load areas of the dataset selectively for processing or analysis.

Tiles represent .las file locations, one file is loaded. Note that the tiles are regular rectangles, and don't always exactly match the extents of the .las file.

[wptabtitle] LP360 tiling tools[/wptabtitle] [wptabcontent]LP360, like most ALS software, provides tools to perform the tiling task. Typical tile sizes include 0.5×0.5km or 1x1km tiles. The naming convention for the tiles should follow a sensible progression, for example reflecting official map grid designations for the area, or following an east to west progressive sequence across the survey area.

The LP360 .las subsetting tool

[wptabtitle] Creating Footprints[/wptabtitle] [wptabcontent]A vector file containing the footprints for each tile, designating the area covered and linking to the .las file or derived terrain models, are a common way of efficiently representing the ALS dataset in a GIS environment. Using LP360, individual files or groups of files can be loaded by selecting their footprints.

Las file footprints are outlined in dark blue; a selected footprint is highlighted.

[wptabtitle] Metadata[/wptabtitle] [wptabcontent]Metadata for ALS is typically generated for the entire survey, rather than per tile. This project level metadata is usually stored in a long form report. That said, some metadata will be stored in the .las header for each tile. Attributes including the total number of points in the file, whether or not it has been classified, and the software used to process the data are typical items found in the header. Further, non-standard, metadata can be stored as a series of attributes in the vector footprint for each .las tile.

Project level metadata provided by the vendor, Aeroquest, provides important information about the survey.

[wptabtitle] laz compression[/wptabtitle] [wptabcontent]The ASPRS standard .las format is commonly used for storing ALS data. The compressed .laz format is also useful, particularly for the datasets which are being archived. Data can be converted from .las to .laz (and back) using LASzip.
[/wptabcontent]
[/wptabs]

Detail from the model of the K11 kerbstone, showing decorative carving on the rock surface.

The k15 kerbstone is available here as a sample dataset. You can download the 3D pdf (low res) or the DAE file (high res).

Photogrammetry data from this project was processed using PhotoScan Pro.

Photoscan Pro processing of the model for the K15 kerbstone.

View Knowth TLS and Photogrammetry in a larger map

NOTE: Even though we are downloading the “Java” Eclipse IDE you can easily add plugins to use this same piece of software with Python, C/C++, and many other applications.

Additionally, we are assuming that you have already gone through the OpenNI installation located here.

You also need to have the Java JDK installed (www.oracle.com).

Finally, to gain access to one of the best open-source computer vision libraries available, you will need to download and install OpenCV(http://opencv.org/) and the JavaCV(http://code.google.com/p/javacv/). The installation instructions located on each of these sites are excellent.
[/wptabcontent]

[wptabtitle] Setting Up Eclipse with OpenNI: Before You Start[/wptabtitle] [wptabcontent]

Important Note: As you may already be aware, these tutorials are focused on the Beginner Level user, not only to using the Kinect but also to programming. Before going any further I should also remind you that if jumping “head first” into the new domain of programming isn’t something for which you have the interest or the time, there are many things you can accomplish with the “ready to use software” solutions located here.

Also, before starting, make sure that you are using the same platform (32 –bit to 32-bit/64 to 64) on the Eclipse IDE, Java JDK, and OpenNI installation.

[/wptabcontent]

[wptabtitle] Eclipse with OpenNI: Starting a New Java Project [/wptabtitle] [wptabcontent]

Starting a New Java Project …..
Once you have downloaded Eclipse, installed the java JDK and the OpenNI/Primesense, you will need to start a new Java Project. Following the wizard is the easiest way to do this.

Check the box that says “public static void main(String[] args)” so that Eclipse will add a few lines of code for us.

NOTE: For this tutorial I have kept the names fairly vague – be sure to use names that you will remember and understand. Remember that if you use a different naming convention than shown here, you will need to make corrections in the sample code to fit to your specifications.

[/wptabcontent]

[wptabtitle] Eclipse with OpenNI: Adding the OpenNI Libraries Part 1[/wptabtitle] [wptabcontent]Adding the OpenNI libraries…

Next we will need to add the OpenNI libraries to the project. This is a pretty straight forward process in Java and Eclipse, simply being a matter of adding the pre-compiled JAR file from the “bin” folder of your OpenNI installation directory.

NOTE: If you plan on using User Tracking or another Primesense middleware capability you will need to add the JAR in the Primesense directory

To do so right-click on the project we just created:

And select the “Properties” menu item.
Then we will want to select the “Java Build Path” and “Add External Jar’s” button.

Repeat the same steps as above for the JavaCV JAR’s that you previously installed somewhere on your machine.

[/wptabcontent]

[wptabtitle] Eclipse with OpenNI: Adding the OpenNI Libraries Part 2[/wptabtitle] [wptabcontent]

Navigate to the “bin” folders of the install directories for OpenNI and Primesense.
On my Windows 7 64-bit machine with the 32-bit install it is located here:

Note: There are TWO OpenNI “JAR” files – one in the bin folder of the OpenNI install directory as well as one in the Primesense directory. I haven’t noticed any difference in using one over the other; as long as your environment paths in Windows are set up to locate the needed files, they should both work.

After this, you should see these files in the “Referenced Libraries” directory on the “Package Explorer” tool bar in Eclipse.

[/wptabcontent]

[wptabtitle] Eclipse with OpenNI: Projects [/wptabtitle] [wptabcontent]

We should now be able to access the Kinect via Java and Eclipse.

In the following projects we will introduce and attempt to explain the necessary steps for initializing the Kinect via OpenNI and for getting basic access to its data feeds in Java.

Each project goes through setting up the Kinect in OpenNI and includes comments to explain line-by-line what is going on.
[/wptabcontent]

[wptabtitle] For information on Visual Studio 2010 & Microsoft C# SDK….[/wptabtitle] [wptabcontent]Using the Microsoft SDK provides a lot of advantages and ease of access, but it is also only applicable to the “Kinect for Windows” hardware and not the Xbox Kinect (as of v1.5).

There are a lot of existing tutorials on the web about setting up your development environment with plenty of sample projects. Below is a list of links to a few of them in no particular order as to avoid reinventing the wheel.

1. http://channel9.msdn.com/Series/KinectQuickstart/Setting-up-your-Development-Environment
2. http://social.msdn.microsoft.com/Forums/el/kinectsdk/thread/7011aca7-defd-445a-bd3c-66837ccc716c
3. http://msdn.microsoft.com/en-us/library/hh855356.aspx
4. Power Point from Stanford

[/wptabcontent]

[/wptabs]

Since there is a plethora of existing tutorials regarding setting up various development environments in C++ and corresponding example projects, this article will show you how to setup the 32-bit OpenNI JAR (OpenNI Java wrapper) in Eclipse IDE. We will then initialize an OpenNI production node to begin accessing Kinect data and to get the RGB stream into OpenCv, which is a popular computer vision library.

Before going on to the following project, make sure that you have all of the dependent libraries installed on your machine. For the instructions on getting the 3rd party libraries and for setting up the development environment check out this post.

Also, I want to clarify that this code is merely one solution that I managed to successfully execute. This said, it may have bugs and/or mayb be done more successfully or more easily using a different solution. If you have any suggestions or find errors, please don’t hesitate to contact us and I will change the post immediately. These posts follow and continue to follow exploration and collaboration.

[/wptabcontent]

[wptabtitle] Using the Kinect’s RGB feed[/wptabtitle] [wptabcontent]

[/wptabcontent]

In this project we will:

1. Make a simple program to capture the RGB feed from the Kinect in Java
2. Get the data into an OpenCV image data structure
3. Display the data on the screen

A high-level overview of the steps we need to take are as follows:

1. Create a new ‘context’ for the Kinect to be started
2. Create and start a ‘generator’ which acts as the mechanism for delivering both data and metadata about its corresponding feed
3. Translate the raw Kinect data into a Java data structure to use in native Java libraries
4. Capture a “frame” and display it on screen

The next tab is the commented code for you to use as you wish.

NOTE: For extremely in-depth and excellent instruction on using JavaCV, the Kinect, along with various other related projects I extremely the book(s) by Andrew Davison from the Imperial College London. A list of his works can be found here http://www.doc.ic.ac.uk/~ajd/publications.html and here http://fivedots.coe.psu.ac.th/~ad/.

[wptabtitle] Sample RGB Project – Part 1[/wptabtitle] [wptabcontent]

[/wptabcontent]

First, let’s import the required libraries:


import org.OpenNI.*;

import com.googlecode.javacv.*;
import static com.googlecode.javacv.cpp.opencv_imgproc.*;
import com.googlecode.javacv.cpp.opencv_core.IplImage;

Then eclipse nicely fills out our class information.


public class sampleRGB {

We define some global variables



static int imWidth, imHeight;

static ImageGenerator imageGen;
static Context contex

Eclipse will also fill out our “main” statement for use by checking the box on the project set up. One addition we will need to make is to surround the following code block with a statement for any exceptions that may be thrown when starting the data feed from the Kinect. Here we are starting a new “context” for the Kinect


public static void main(String[] args) throws GeneralException {
Create a “context”


context = new Context();

[wptabtitle] Sample RGB Project – Part 2[/wptabtitle] [wptabcontent]

[/wptabcontent]

We are manually adding the license information from Primesense. You can also directly reference the xml documents located in the install directory of both the OpenNI and Primesense.


License license = new License("PrimeSense", "0KOIk2JeIBYClPWVnMoRKn5cdY4=");
context.addLicense(license);


Create a “generator” which is the machine that will pump out RGB data



imageGen = ImageGenerator.create(context);


We need to define the resolution of the data coming from the image generator. OpenNI calls this mapmode (imageMaps, depthMaps, etc.). We will use the standard resolution.

First initialize it to null.



MapOutputMode mapMode = null;
mapMode = new MapOutputMode(640, 480, 30);
imageGen.setMapOutputMode(mapMode);


We also need to pick the pixel format to display from the Image Generator. We will use the Red-Green-Blue 8-bit 3 channel or “RGB24”



imageGen.setPixelFormat(PixelFormat.RGB24);

[wptabtitle] Sample RGB Project – Part 3[/wptabtitle] [wptabcontent]

[/wptabcontent]

OpenNI also allows us to easily mirror the image so movement in 3d space is reflected in the image plane



context.setGlobalMirror(true);


Create an Iplimage(opencv image) with the same size and format as the feed from the kinect.



IplImage rgbImage = IplImage.create(imWidth,imHeight, 8, 3);


Next we will use the easy route and utilize JFrame/Javacv optimized canvas to show the image


CanvasFrame canvas = new CanvasFrame("RGB Demo");


Now we will create a never-ending loop to update the data and frames being displayed on the screen. Going line by line we will, update the context every time the image generator gets new data.
Set the opencv image data to the byte buffer created from the imageGen.

NOTE: For some reason the channels coming from the Kinect to the opencv image are ordered differently so we will simply use the opencv convert color to set the “BGR” to “RGB”. We tell the canvas frame that we created to show image.

[wptabtitle] Sample RGB Project – Part 4[/wptabtitle] [wptabcontent]

[/wptabcontent]

Finally, we need to also release the Kinect context or we will get an error the next time we try to start a node because the needed files will be locked


while (true){
context.waitOneUpdateAll(imageGen);
rgbImage.imageData(imageGen.createDataByteBuffer());
cvCvtColor(rgbImage, rgbImage, CV_BGR2RGB);
canvas.showImage(rgbImage);

canvas.setDefaultCloseOperation(CanvasFrame.EXIT_ON_CLOSE);
}

**Note that you can add an argument to the canvas frame to reverse the channels of the image

[/wptabs]

[wptabtitle] RGB Image[/wptabtitle]

[wptabcontent]The color image streaming from the RGB camera is much like that from your average webcam. It has a standard resolution of Height:640 Width:480 at a frame rate of 30 frames per second. You can “force” the Kinect to output a higher resolution image (1280×960) but it will significantly reduce it’s frame rate.

Many things can be done with the RGB data alone such as:

• Image or Video Capture
• Optical Flow Tracking
• Capturing data for textures of models
• Facial Recognition
• Motion Tracking
• And many more….

While it seems silly to purchase a Kinect (about $150) just to use it as a webcam – it is possible. In fact there are ways to hook the camera up to Microsoft’s DirectShow to use it with Skype and other webcam-enabled programs. (Check out this project http://www.e2esoft.cn/kinect/)

• For an Example Project Using the Kinect RGB feed ……

[/wptabcontent]

[wptabtitle] Depth Image[/wptabtitle]

[wptabcontent] The Kinect is suited with two pieces of hardware, which through their combined efforts, give us the “Depth Image”.
It is the Infrared projector with the CMOS IR “camera” that measures the “distance” from the sensor to the corresponding object off of which the Infrared light reflects.

I say “distance” because the depth sensor of the Kinect actually measures the time that the light takes to leave the sensor and to return to the camera. The returning signal to the Kinect can be altered by other factors including:

• The physical distance – the return of this light is dependent on it reflecting off of an object within the range of the Kinect (~1.2–3.5m)
• The surface – like other similar technology (range cameras, laser scanners, etc.) the surface which the IR beam hits affects the returning signal. Most commonly glossy or highly reflective, screens (TV,computer,etc.), and windows pose issues for receiving accurate readings from the sensor.

[/wptabcontent]

[wptabtitle] The IR Projector[/wptabtitle]

[wptabcontent]

The IR projector does not emit uniform beams of light but instead relies on a “speckle pattern” according to the U.S. Patent (located here:http://www.freepatentsonline.com/7433024.pdf )

You can actually see the IRMap, as it’s called, using OpenNI. Here is a picture from Matthew Fisher’s website and another excellent resource on the Kinect (http://graphics.stanford.edu/~mdfisher/Kinect.html)

The algorithm used to compute the depth by the Kinect is derived from the difference between the speckle pattern that is observed and a reference pattern at a known depth.

“The depth computation algorithm is a region-growing stereo method that first finds anchor points using normalized correlation and then grows the solution outward from these anchor points using the assumption that the depth does not change much within a region.”

For a deeper discussion on the IR pattern from the Kinect check out this site : http://www.futurepicture.org/?p=116

[/wptabcontent]

[wptabtitle] Coordinate System[/wptabtitle]

[wptabcontent]

As one might assume the Kinect uses these “anchor points” as references in it’s own internal coordinate system. This origin coordinate system is shown here:

So the (x) is the to left of the sensor, (y) is up, and (z) is going out away from the sensor.

This is shown by the (Kx,Ky,Kz) in the above image, with the translated real world coordinates as (Sx,Sy,Sz).

The image is in meters (to millimeter precision).

[/wptabcontent]

[wptabtitle] Translating Depth & Coordinates[/wptabtitle]

[wptabcontent]

So when you work with the Depth Image and plan on using it to track, identify, or measure objects in real world coordinates you will have to translate the pixel coordinate to 3D space.OpenNI makes this easy by using their function (XnStatus xn::DepthGenerator::ConvertProjectiveToRealWorld) which converts a list of points from projective(internal Kinect) coordinates to real world coordinates.

Of course, you can go the other way too, taking real world coordinates to the projective using (XnStatus xn::DepthGenerator::ConvertRealWorldToProjective)

The depth feed from the sensor is 11-bits, therefore, it is capable of capturing a range of 2,048 values. In order to display this image in more 8-bit image structures you will have to convert the range of values into a 255 monochromatic scale. While it is possible to work with what is called the “raw” depth feed in some computer vision libraries(like OpenCV) most of the examples I’ve seen convert the raw depth feed in the same manner. That is to create a histogram from the raw data and to assign the corresponding depth value (from 0 to 2,048) to one of the 255 “bins” which will be the gray-scale value of black to white(0-255) in an 8-bit monochrome image.

You can look at the samples given by OpenNI to get the code, which can be seen in multiple programming languages by looking at their “Viewer” samples.

[/wptabcontent]

[wptabtitle] Accuracy[/wptabtitle]

[wptabcontent]

Another thing worth noting is the difference in accuracy of the depth image as distance from the Kinect increases. There seems to be a decrease in accuracy as one gets further away from the sensor, which makes sense when looking at the previous image of the pattern that the IR projector emits. The greater the physical distance between the object and the IR projector, the less coverage the speckle pattern has on that object in-between anchor points. Or in other words the dots are spaced further apart(x,y) as distance(z) is increased.

The Kinect comes factory calibrated and according to some sources, it isn’t that far off for most applications.

[/wptabcontent]

[wptabtitle] Links for Recalibrating[/wptabtitle]

[wptabcontent]
Here are some useful links to recalibrating the Kinect if you want to learn more:

• http://www.ros.org/news/2010/12/technical-information-on-kinect-calibration.html
• http://labs.manctl.com/rgbdemo/index.php/Documentation/TutorialProjectorKinectCalibration
• http://openkinect.org/wiki/Calibration
• http://nicolas.burrus.name/index.php/Research/KinectCalibration
• http://sourceforge.net/projects/kinectcalib/ (a MatLab ToolBox)

[/wptabcontent]

[wptabtitle] User Tracking[/wptabtitle]

[wptabcontent]
The Kinect comes with the capability to track users movements and to identify several joints of each user being tracked. The applications that this kind of readily accessible information can be applied to are plentiful. The basic capabilities of this feature, called “skeletal tracking”, have extended further for pose detection, movement prediction, etc.

According to information supplied to retailers, Kinect is capable of simultaneously tracking up to six people, including two active players, for motion analysis with a feature extraction of 20 joints per player. However,PrimeSense has stated that the number of people that the device can “see” (but not process as players) is only limited by how many will fit into the field-of-view of the camera.

Tracking 2 users **Microsoft

An in depth explanation can be found on the patent application for the Kinect here:http://www.engadget.com/photos/microsofts-kinect-patent-application/

[/wptabcontent]

[wptabtitle] Point Cloud Data[/wptabtitle]

[wptabcontent]

The Kinect doesn’t actually capture a ‘point cloud’. Rather you can create one by utilizing the depth image that the IR sensor creates. Using the pixel coordinates and (z) values of this image you can transform the stream of data into a 3D “point cloud”. Using an RGB image feed and a depth map combined, it is possible to project the colored pixels into three dimensions and to create a textured point cloud.

Instead of using 2D graphics to make a depth or range image, we can apply that same data to actually position the “pixels” of the image plane to 3D space. This allows one to view objects from different angles and lighting conditions. One of the advantages of transforming data into a point cloud structure is that it provides for more robust analysis and for more dynamic use than the same data in the form of a 2D graphic.

Connecting this to the geospatial world can be analogous to the practice of extruding Digital Elevation Models (DEM’s) of surface features to three dimensions in order to better understand visibility relationships, slope, environmental dynamics, and distance relationships. While it is certainly possible to determine these things without creating a point cloud, the added ease of interpreting these various relationships from data in a 3D format is self-evident and inherent. Furthermore, the creation of a point cloud allows for an easy transition to creating 3D models that can be applied to various domains from gaming to planning applications.

So with two captured images like this:

Depth Map of the imaged scene, shown left in greyscale.

We can create a 3D point cloud.

[/wptabcontent]

[wptabtitle] Setting Up Your Development Environment[/wptabtitle]

[wptabcontent] We will give you a few examples on how to set up your Development Environment using Kinect API’s.

These are all going to be demonstrated on a Windows 7 64-bit machine using only the 32-bit versions of the downloads covered here.

At the time of this post the versions we will be using are:
OpenNI: v 1.5.2.23
Microsoft SDK: v 1.5
OpenKinect(libfreenect): Not Being Done at this time… Sorry

To use the following posts you need to have installed the above using these directions.

• For Information on setting up Eclipse and the OpenNI/Primesense Java ….

[/wptabcontent]
[/wptabs]

Resources and Learning

1.   www.kinecthacks.com

2.   www.kinect.dashhacks.com

3.   www.kinecteducation.com

4.   www.developkinect.com

5.   www.scratch.saorog.com

6.   www.microsoft.com/education/ww/partners-in-learning/Pages/index.aspx

7.  blogs.msdn.com/b/uk_faculty_connection/archive/2012/04/21/kinect-for-windows-curriculum

8.   dotnet.dzone.com/articles/kinect-sdk-resources

9.   hackaday.com/2012/03/22/kinect-for-windows-resources

10. channel9.msdn.com/coding4fun/kinect

11. www.pcworld.com/article/217283/top_15_kinect_hacks_so_far.html

[/wptabcontent]

[wptabtitle] Links: OpenNI[/wptabtitle] [wptabcontent]

OpenNI

• openni.org – Open Natural Interaction, an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware

• github.com/openni – Open source framework for natural interaction devices

• github.com/PrimeSense/Sensor – Open source driver for the PrimeSensor Development Kit

[/wptabcontent]

[wptabtitle] Links: Tech[/wptabtitle] [wptabcontent]

Tech

1. www.ifixit.com/Teardown/Microsoft-Kinect-Teardown/4066 – Hardware teardown. Chip info is here. (via adafruit)

2. kinecthacks.net/kinect-pinout – Pinout info of the Kinect Sensor

3. www.primesense.com/?p=535 – Primesense reference implementation (via adafruit thread)

4. www.sensorland.com/HowPage090.html – How sensors work and the bayer filter

5. www.numenta.com/htm-overview/education/HTM_CorticalLearningAlgorithms.pdf – Suggestions to implement pseudocode near the end

6. http://www.dwheeler.com/essays/floss-license-slide.html – Which licenses are compatible with which

7. http://www.eetimes.com/design/signal-processing-dsp/4211071/Inside-Xbox-360-s-Kinect-controller – Another Hardware Teardown. Note this article incorrectly states that the PS1080 talks to the Marvell chip.

8. http://nvie.com/posts/a-successful-git-branching-model/ – Model for branching within Git

9. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/SubmittingPatches – Linux contribution procedure

10. http://git.kernel.org/?p=git/git.git;a=blob_plain;f=Documentation/SubmittingPatches;hb=HEAD – Git project contribution procedure

[/wptabcontent]

[wptabtitle] Hardware Options[/wptabtitle]

[wptabcontent]

Well the first option is to build your own, here’s a how-to:

http://www.hackengineer.com/3dcam/

But since not all of us have the time of skills to do that there are other options, like….

1- ASUS Xtion PRO:
Price: $140
Spec’s: http://www.newegg.com/Product/Product.aspx?Item=N82E16826785030

2- Leap Motion:
Price: $70
Spec’s: https://live.leapmotion.com/about.html

Of course there may be more and there is talk of Sony recently filling a patent resembling their own “Kinect-like” device.

[/wptabcontent]

[wptabtitle] Xbox Kinect vs. Kinect for Windows[/wptabtitle]

[wptabcontent]

As you may know there are actually two “Kinect” sensors out on the market today…

both under the Microsoft company but one was the original made for the Xbox 360 Game console, while the other is the recently released “Kinect for Windows”.

Overview

As far as I can tell the two hardware stacks are identical except for the name plate on the front (XBOX or KINECT FOR WINDOWS) and the Windows version has a shorter power cord with a higher price tag due to licensing issues.

There are constant changes being done to the Kinect for Windows to distance itself from it’s Xbox twin, like a firmware update to support “Near Mode” in the Windows SDK….

Microsoft even goes as far as saying the following:

‘The Kinect for Windows SDK has been designed for the Kinect for Windows hardware and application development is only licensed with use of the Kinect for Windows sensor. We do not recommend using Kinect for Xbox 360 to assist in the development of Kinect for Windows applications. Developers should plan to transition to Kinect for Windows hardware for development purposes and should expect that their users will also be using Kinect for Windows hardware as well.’

If you are currently using the Kinect for Xbox you will find that the automatic registration functions found with the Microsoft SDK will not recognize your Kinect and therefore kick out an error every time you try to run one of their samples.

As far as I know, you can however, still manually register the Kinect with the Microsoft SDK and utilize the functions already developed in the API AT THIS POINT with the XBOX version of the sensor. I wouldn’t be surprised if this changes in the near future however.

[/wptabcontent]

[wptabtitle] Published Resources[/wptabtitle] [wptabcontent]

Published Resources

1) Abramov, Alexey et al. “Depth-supported Real-time Video Segmentation with the Kinect.” Proceedings of the 2012 IEEE Workshop on the Applications of Computer Vision. Washington, DC, USA: IEEE Computer Society, 2012. 457–464. Web. 6 July 2012. WACV ’12.

2) Bleiweiss, Amit et al. “Enhanced Interactive Gaming by Blending Full-body Tracking and Gesture Animation.” ACM SIGGRAPH ASIA 2010 Sketches. New York, NY, USA: ACM, 2010. 34:1–34:2. Web. 6 July 2012. SA ’10.

3) Borenstein, Greg. Making Things See: 3D Vision with Kinect, Processing, Arduino, and MakerBot. Make, 2012. Print.

4) Boulos, Maged N Kamel et al. INTERNATIONAL JOURNAL OF HEALTH GEOGRAPHICS EDITORIAL Open Access Web GIS in Practice X: a Microsoft Kinect Natural User Interface for Google Earth Navigation. Print.

5) Burba, Nathan et al. “Unobtrusive Measurement of Subtle Nonverbal Behaviors with the Microsoft Kinect.” Proceedings of the 2012 IEEE Virtual Reality. Washington, DC, USA: IEEE Computer Society, 2012. 1–4. Web. 6 July 2012. VR ’12.

6) Center for History and New Media. “Zotero Quick Start Guide.”

7) Clark, Adrian, and Thammathip Piumsomboon. “A Realistic Augmented Reality Racing Game Using a Depth-sensing Camera.” Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry. New York, NY, USA: ACM, 2011. 499–502. Web. 6 July 2012. VRCAI ’11.

8 ) Cui, Yan, and Didier Stricker. “3D Shape Scanning with a Kinect.” ACM SIGGRAPH 2011 Posters. New York, NY, USA: ACM, 2011. 57:1–57:1. Web. 6 July 2012. SIGGRAPH ’11.

9) Davison, Andrew. Kinect Open Source Programming Secrets: Hacking the Kinect with OpenNI, NITE, and Java. 1st ed. McGraw-Hill/TAB Electronics, 2012. Print.

10) Devereux, D. et al. “Using the Microsoft Kinect to Model the Environment of an Anthropomimetic Robot.” Submitted to the Second IASTED International Conference on Robotics (ROBO 2011). Web. 6 July 2012.

11) Dippon, Andreas, and Gudrun Klinker. “KinectTouch: Accuracy Test for a Very Low-cost 2.5D Multitouch Tracking System.” Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. New York, NY, USA: ACM, 2011. 49–52. Web. 6 July 2012. ITS ’11.

12) Droeschel, David, and Sven Behnke. “3D Body Pose Estimation Using an Adaptive Person Model for Articulated ICP.” Proceedings of the 4th International Conference on Intelligent Robotics and Applications – Volume Part II. Berlin, Heidelberg: Springer-Verlag, 2011. 157–167. Web. 6 July 2012. ICIRA’11.

13) Dutta, Tilak. “Evaluation of the Kinect Sensor for 3-D Kinematic Measurement in the Workplace.” Applied Ergonomics 43.4 (2012): 645–649. Web. 6 July 2012.

14) Engelharda, N. et al. “Real-time 3D Visual SLAM with a Hand-held RGB-D Camera.” Proc. of the RGB-D Workshop on 3D Perception in Robotics at the European Robotics Forum, Vasteras, Sweden. Vol. 2011. 2011. Web. 6 July 2012.

15) Francese, Rita, Ignazio Passero, and Genoveffa Tortora. “Wiimote and Kinect: Gestural User Interfaces Add a Natural Third Dimension to HCI.” Proceedings of the International Working Conference on Advanced Visual Interfaces. New York, NY, USA: ACM, 2012. 116–123. Web. 6 July 2012. AVI ’12.

16) Giles, J. “Inside the Race to Hack the Kinect.” The New Scientist 208.2789 (2010): 22–23. Print.

17) Gill, T. et al. “A System for Change Detection and Human Recognition in Voxel Space Using the Microsoft Kinect Sensor.” Proceedings of the 2011 IEEE Applied Imagery Pattern Recognition Workshop. Washington, DC, USA: IEEE Computer Society, 2011. 1–8. Web. 6 July 2012. AIPR ’11.

18) Gomez, Juan Diego et al. “Toward 3D Scene Understanding via Audio-description: Kinect-iPad Fusion for the Visually Impaired.” The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. New York, NY, USA: ACM, 2011. 293–294. Web. 6 July 2012. ASSETS ’11.

19) Goth, Gregory. “Brave NUI World.” Commun. ACM 54.12 (2011): 14–16. Web. 6 July 2012.

20) Gottfried, Jens-Malte, Janis Fehr, and Christoph S. Garbe. “Computing Range Flow from Multi-modal Kinect Data.” Proceedings of the 7th International Conference on Advances in Visual Computing – Volume Part I. Berlin, Heidelberg: Springer-Verlag, 2011. 758–767. Web. 6 July 2012. ISVC’11.

21) Henry, P. et al. “RGB-D Mapping: Using Kinect-style Depth Cameras for Dense 3D Modeling of Indoor Environments.” The International Journal of Robotics Research (2012): n. pag. Web. 6 July 2012.

22) Henry, Peter et al. “RGB-D Mapping: Using Kinect-style Depth Cameras for Dense 3D Modeling of Indoor Environments.” Int. J. Rob. Res. 31.5 (2012): 647–663. Web. 6 July 2012.

23) Hilliges, Otmar et al. “HoloDesk: Direct 3d Interactions with a Situated See-through Display.” Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems. New York, NY, USA: ACM, 2012. 2421–2430. Web. 6 July 2012. CHI ’12.

24) Izadi, Shahram et al. “KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera.” Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. New York, NY, USA: ACM, 2011. 559–568. Web. 6 July 2012. UIST ’11.

25) Jean, Jared St. Kinect Hacks: Creative Coding Techniques for Motion and Pattern Detection. O’Reilly Media, 2012. Print.

26) Kean, Sean, Jonathan Hall, and Phoenix Perry. Meet the Kinect: An Introduction to Programming Natural User Interfaces. 1st ed. Berkely, CA, USA: Apress, 2011. Print.

27) Kramer, Jeff et al. Hacking the Kinect. 1st ed. Berkely, CA, USA: Apress, 2012. Print.

28) LaViola, Joseph J., and Daniel F. Keefe. “3D Spatial Interaction: Applications for Art, Design, and Science.” ACM SIGGRAPH 2011 Courses. New York, NY, USA: ACM, 2011. 1:1–1:75. Web. 6 July 2012. SIGGRAPH ’11.

29) Li, Li, Yanhao Xu, and Andreas König. “Robust Depth Camera Based Eye Localization for Human-machine Interactions.” Proceedings of the 15th International Conference on Knowledge-based and Intelligent Information and Engineering Systems – Volume Part I. Berlin, Heidelberg: Springer-Verlag, 2011. 424–435. Web. 6 July 2012. KES’11.

30) Livingston, Mark A. et al. “Performance Measurements for the Microsoft Kinect Skeleton.” Proceedings of the 2012 IEEE Virtual Reality. Washington, DC, USA: IEEE Computer Society, 2012. 119–120. Web. 6 July 2012. VR ’12.

31) Melgar, Enrique Ramos, and Ciriaco Castro Diez. Arduino and Kinect Projects: Design, Build, Blow Their Minds. 1st ed. Berkely, CA, USA: Apress, 2012. Print.

32) Miles, Helen C. et al. “A Review of Virtual Environments for Training in Ball Sports.” Computers & Graphics 36.6 (2012): 714–726. Web. 6 July 2012.

33) Miles, Rob. Start Here! Learn the Kinect API. Microsoft Press, 2012. Print.

34) Mitchell, Grethe, and Andy Clarke. “Capturing and Visualising Playground Games and Performance: a Wii and Kinect Based Motion Capture System.” Proceedings of the 2011 International Conference on Electronic Visualisation and the Arts. Swinton, UK, UK: British Computer Society, 2011. 218–225. Web. 6 July 2012. EVA’11.

35) Molyneaux, David. “KinectFusion Rapid 3D Reconstruction and Interaction with Microsoft Kinect.” Proceedings of the International Conference on the Foundations of Digital Games. New York, NY, USA: ACM, 2012. 3–3. Web. 6 July 2012. FDG ’12.

36) Mutto, Carlo Dal, Pietro Zanuttigh, and Guido M. Cortelazzo. Time-of-Flight Cameras and Microsoft Kinect(TM). Springer Publishing Company, Incorporated, 2012. Print.

37) Panger, Galen. “Kinect in the Kitchen: Testing Depth Camera Interactions in Practical Home Environments.” Proceedings of the 2012 ACM Annual Conference Extended Abstracts on Human Factors in Computing Systems Extended Abstracts. New York, NY, USA: ACM, 2012. 1985–1990. Web. 6 July 2012. CHI EA ’12.

38) Pheatt, Chuck, and Jeremiah McMullen. “Programming for the Xbox Kinect Sensor: Tutorial Presentation.” J. Comput. Sci. Coll. 27.5 (2012): 140–141. Print.

39) Raheja, Jagdish L., Ankit Chaudhary, and Kunal Singal. “Tracking of Fingertips and Centers of Palm Using KINECT.” Proceedings of the 2011 Third International Conference on Computational Intelligence, Modelling & Simulation. Washington, DC, USA: IEEE Computer Society, 2011. 248–252. Web. 6 July 2012. CIMSIM ’11.

40) Riche, Nicolas et al. “3D Saliency for Abnormal Motion Selection: The Role of the Depth Map.” Proceedings of the 8th International Conference on Computer Vision Systems. Berlin, Heidelberg: Springer-Verlag, 2011. 143–152. Web. 6 July 2012. ICVS’11.

41) Rogers, Rick. “Kinect with Linux.” Linux J. 2011.207 (2011): n. pag. Web. 6 July 2012.

42) Shrewsbury, Brandon T. “Providing Haptic Feedback Using the Kinect.” The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. New York, NY, USA: ACM, 2011. 321–322. Web. 6 July 2012. ASSETS ’11.

43) Sidik, Mohd Kufaisal bin Mohd et al. “A Study on Natural Interaction for Human Body Motion Using Depth Image Data.” Proceedings of the 2011 Workshop on Digital Media and Digital Content Management. Washington, DC, USA: IEEE Computer Society, 2011. 97–102. Web. 6 July 2012. DMDCM ’11.

44) Smisek, J., M. Jancosek, and T. Pajdla. “3D with Kinect.” Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference On. 2011. 1154 –1160.

45) Solaro, John. “The Kinect Digital Out-of-Box Experience.” Computer 44.6 (2011): 97–99. Web. 6 July 2012.

46) Stone, E. E, and M. Skubic. “Evaluation of an Inexpensive Depth Camera for Passive In-home Fall Risk Assessment.” Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2011 5th International Conference On. 2011. 71–77. Web. 6 July 2012.

47) Sturm, J. et al. “Towards a Benchmark for RGB-D SLAM Evaluation.” Proc. of the RGB-D Workshop on Advanced Reasoning with Depth Cameras at Robotics: Science and Systems Conf.(RSS), Los Angeles, USA. Vol. 2. 2011. 3. Web. 6 July 2012.

48) Sung, J. et al. “Human Activity Detection from RGBD Images.” AAAI Workshop on Pattern, Activity and Intent Recognition (PAIR). 2011. Web. 6 July 2012.

49) Tang, John C., Carolyn Wei, and Reena Kawal. “Social Telepresence Bakeoff: Skype Group Video Calling, Google+ Hangouts, and Microsoft Avatar Kinect.” Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work Companion. New York, NY, USA: ACM, 2012. 37–40. Web. 6 July 2012. CSCW ’12.

50) “The Kinect Revolution.” The New Scientist 208.2789 (2010): 5. Web. 6 July 2012.

51) Tong, Jing et al. “Scanning 3D Full Human Bodies Using Kinects.” IEEE Transactions on Visualization and Computer Graphics 18.4 (2012): 643–650. Web. 6 July 2012.

52) Villaroman, Norman, Dale Rowe, and Bret Swan. “Teaching Natural User Interaction Using OpenNI and the Microsoft Kinect Sensor.” Proceedings of the 2011 Conference on Information Technology Education. New York, NY, USA: ACM, 2011. 227–232. Web. 6 July 2012. SIGITE ’11.

53) “Virtual Reality from the Keyboard/mouse Couple to Kinect.” Annals of Physical and Rehabilitation Medicine 54, Supplement 1.0 (2011): e239. Web. 6 July 2012.

54) Webb, Jarrett, and James Ashley. Beginning Kinect Programming with the Microsoft Kinect SDK. 1st ed. Apress, 2012. Print.

55) Weise, Thibaut et al. “Kinect-based Facial Animation.” SIGGRAPH Asia 2011 Emerging Technologies. New York, NY, USA: ACM, 2011. 1:1–1:1. Web. 6 July 2012. SA ’11.

56) Wilson, Andrew D. Using a Depth Camera as a Touch Sensor. Print.

57) Xu, Yunfei, and Jongeun Choi. “Spatial Prediction with Mobile Sensor Networks Using Gaussian Processes with Built-in Gaussian Markov Random Fields.” Automatica 0 n. pag. Web. 6 July 2012.

58) Zhang, Zhengyou. “Microsoft Kinect Sensor and Its Effect.” IEEE MultiMedia 19.2 (2012): 4–10. Web. 6 July 2012.

[/wptabcontent]

[/wptabs]

[wptabtitle] Getting Started Fast with the Kinect [/wptabtitle]

[wptabcontent]

Getting Started

There are a variety of software that are being or have been developed that provide you with access to the Kinect sensor through interactive GUI’s. Some use different API’s and libraries (OpenNI,Microsoft SDK, etc.) to do this communication with the Kinect and these may conflict with software that you have already installed. So read the sites before downloading and installing. Make sure that you do clean installs. 

In this post we will provide you with a high level introduction to two excellent software suites provided on a free-to-use basis. 

Instant Access

These provide instant access to the sensor data and help to get your project started whether it involves scanning, recording, viewing, or streaming from the Kinect. 

Follow the links at the end of the pages to see an example workflow with each of the two software packages. 
[/wptabcontent]

[wptabtitle]
RGBDemo
[/wptabtitle]

[wptabcontent]

RGB Demo was initially developed by Nicolas Burrus in the RoboticsLab. He then co-founded the Manctl company that now maintains it, helped by various contributors from the opensource community.

Current features

• Grab kinect images and visualize / replay them
• Support for libfreenect and OpenNI/Nite backends
• Extract skeleton data / hand point position (Nite backend)
• Integration with OpenCV and PCL
• Multiple Kinect support and calibration
• Calibrate the camera to get point clouds in metric space (libfreenect)
• Export to meshlab/blender using .ply files
• Demo of 3D scene reconstruction using a freehand Kinect
• Demo of people detection and localization
• Demo of gesture recognition and skeleton tracking using Nite
• Demo of 3D model estimation of objects lying on a table (based on PCL table top object detector)
• Demo of multiple kinect calibration
• Linux, MacOSX and Windows support

[wptabtitle] Brekel Kinect[/wptabtitle]

[wptabcontent]
In my opinion the Brekel Kinect software is the best that I have seen for easily interfacing with the Kinect.

It uses the OpenNI framework, which we show on the GMV here, but the developer of this software also provides their own packaged OpenNI installer that comes with all the required dependencies. 

Current Features

The Brekel Kinect only offers binaries for the Windows platform. It was developed by  Jasper Brekelmans in his free time. 

It allows you to capture 3D objects and to export them to disk for use in 3D packages. It also allows you to do skeleton tracking which can be streamed into Autodesk’s MotionBuilder in realtime, or exported as BVH files.

The greatest things about the Brekel software are that it requires no programming expertise, it has an easy to use GUI, and it has the ability to export almost all of the Kinect’s capabilities in a variety of formats.

The Brekel website offers a bunch of links to other resources, downloads, tutorials and answers to FAQ’s. Check it out at:

Main Site: http://www.brekel.com

[/wptabcontent]

[/wptabs]