ALS data from the Bayou Meto undergoing processing

CAST researchers and student assistants developed a hydro-enforced DTM (Digital Terrain Model) covering the Bayou Meto watershed, in collaboration with the NCRS and Arkansas Natural Resources Commission. The classification of the raw ALS data, interpolation to basic bare-earth terrain models, the creation of breaklines and streamlines for hydro-enforcement, and the refinement of final hydro-enforced models were carried out at CAST.

Data for this project was collected by Aeroquest in 2009 and 2010 for two areas within the Bayou Meto, at a nominal resolution of 10 pts/m2. The TIFFS and LP360 software packages were used to process the discrete return ALS data and to assist in breakline production.

For more information, see the main project webpage.

Hydro-enforced DTMs include breaklines, importantly stream centerlines and edges, and breaklines delimiting standing water bodies such as ponds. While auto-extraction of breaklines is improving, the creation of hydro-enforcing features is still by and large a manual task.

Deliverables include hydro-DTMs.

[wptabtitle] DSMs[/wptabtitle] [wptabcontent]Digital Surface Models (DSMs) can include only returns from the terrain, buildings and specific classes of off-terrain objects like bridges, or can also incorporate returns from vegetation. DSMs are commonly used in urban environment analyses such as noise pollution modeling and inter-visibility analyses to assess the impact of new building.

DSMs are often used for modeling in urban areas.

[wptabtitle] CHM[/wptabtitle] [wptabcontent]Canopy height models, and per-stand or individual tree metrics are important ALS-based products for forestry applications. These models often include returns separated into low- mid- and high- vegetation classes, and are sometimes normalized based on local terrain heights to facilitate comparisons between different forest areas.

Canopy height model generated using SAGA GIS.

[wptabtitle] Contours[/wptabtitle] [wptabcontent]Contour maps at standard intervals, e.g. 1m, 5m, or 20m contours, can be generated from bare earth DTMs. Contour maps can be generated with or without breaklines.

Contours developed based on the terrain model.

Automatically classified Point Cloud seen in profile. Terrain points (class 2) are orange, and off-terrain points are grey.

[wptabtitle] Create DTMs and Hillshades[/wptabtitle] [wptabcontent]To facilitate visually identifying incorrectly classified returns, it’s useful to interpolate the automatically classified ground points into a DTM, and to create basic hillshades. Many classification errors will be readily apparent in the hillshaded models. The DTMs for the Bayou Meto project were created using LP360 for ArcGIS.

Bare earth DTM created before manual re-classification.

[wptabtitle] Linked Viewers[/wptabtitle] [wptabcontent]Viewing the point cloud simultaneously with the hillshaded terrain model, you can navigate quickly to ‘problem areas’ to re-classify any incorrect points in the ALS point cloud.

Linked viewers allow simultaneous viewing as a 3d point cloud, in profile, and at a shaded DTM.

[wptabtitle] Drawing Profiles[/wptabtitle] [wptabcontent]Draw a profile across an area of the terrain model where potential mis-classifications have been identified. Depending on how regular the terrain surface is, set the depth of the profile. Areas where the elevation of the terrain varies greatlygenerally require narrower profiles to clearly visualize the separation between the ground surface and low vegetation.

Drawing the profile on the DTM.

[wptabtitle] Editing classifications[/wptabtitle] [wptabcontent]When editing the classification of the points it’s best to set the point cloud coloring style to ‘by class’ rather than by elevation or by return, as it’s then easier to see which points should be re-classified. In LP360 you can change the classification of points by selecting them in the profile view using a ‘brush’ or ‘lasso’ tool and then typing the number of the class they should be and hitting enter. [/wptabcontent]

[wptabtitle] Typical problem areas[/wptabtitle] [wptabcontent]Work across the dataset systematically, until all problem areas have been improved. Note that areas with dense, low vegetation, large numbers of small buildings, and mixed steep slopes and vegetation are the most likely to contain mis-classified returns, and will require more effort. In the Bayou Meto dataset, the edges of streams proved typical problem areas, combining sloping terrain and low, dense vegetation.

Typical problem area circled in red, located under vegetation at the base of the slope.

[wptabtitle] Re-creating the terrain models[/wptabtitle] [wptabcontent]After re-classifying the ALS returns, it is necessary to re-create the terrain models and any other derivatives. These new models are the basis for further processing and analysis.

Hillshades and other derivatives are created from the cleaned point clouds.

A tie strip can be seen here overlapping with two flightlines.

[wptabtitle] Tile Schemes[/wptabtitle] [wptabcontent]Because ALS datasets are usually very large, they are often divided into regularly sized tiles. These tiling schemes can help with the speed of data loading, and allow users to load areas of the dataset selectively for processing or analysis.

Tiles represent .las file locations, one file is loaded. Note that the tiles are regular rectangles, and don't always exactly match the extents of the .las file.

[wptabtitle] LP360 tiling tools[/wptabtitle] [wptabcontent]LP360, like most ALS software, provides tools to perform the tiling task. Typical tile sizes include 0.5×0.5km or 1x1km tiles. The naming convention for the tiles should follow a sensible progression, for example reflecting official map grid designations for the area, or following an east to west progressive sequence across the survey area.

The LP360 .las subsetting tool

[wptabtitle] Creating Footprints[/wptabtitle] [wptabcontent]A vector file containing the footprints for each tile, designating the area covered and linking to the .las file or derived terrain models, are a common way of efficiently representing the ALS dataset in a GIS environment. Using LP360, individual files or groups of files can be loaded by selecting their footprints.

Las file footprints are outlined in dark blue; a selected footprint is highlighted.

[wptabtitle] Metadata[/wptabtitle] [wptabcontent]Metadata for ALS is typically generated for the entire survey, rather than per tile. This project level metadata is usually stored in a long form report. That said, some metadata will be stored in the .las header for each tile. Attributes including the total number of points in the file, whether or not it has been classified, and the software used to process the data are typical items found in the header. Further, non-standard, metadata can be stored as a series of attributes in the vector footprint for each .las tile.

Project level metadata provided by the vendor, Aeroquest, provides important information about the survey.

[wptabtitle] laz compression[/wptabtitle] [wptabcontent]The ASPRS standard .las format is commonly used for storing ALS data. The compressed .laz format is also useful, particularly for the datasets which are being archived. Data can be converted from .las to .laz (and back) using LASzip.
[/wptabcontent]
[/wptabs]

Many archaeological projects use a GIS to manage their data. After terrestrial scan or photogrammetric modeling data has been collected and cleaned, it may be convenient to integrate it into a project’s GIS setup. As ArcGIS is widely available and in use both in University research departments and government offices, we’re using it for the example here, but something like this should work for other GIS packages.

The first part of the workflow addresses working with meshes created from terrestrial scan data, and assumes you have existing meshes in Rapidform.

[/wptabcontent]

[wptabtitle] Decimation[/wptabtitle] [wptabcontent]

Before exporting a dataset for use in a GIS you may want to decimate the dataset to produce a lower resolution model for visualization. High resolution models can slow rendering down and make manipulation of the model difficult.

a. Select the model you will be exporting either graphically or through the menu tree on the left hand side of the screen.

b. In the main menu select Tools and then Scan Tools and Decimate Meshes

Fig. 1: Select the Decimate Meshes tool

c. In the Decimate Meshes menu confirm the selection of the Target Mesh.

d. Under Method choose Poly-Face Count for best control over the size of the resultant model.

e. Under Options set the Target Poly-Face Count. Numbers under 100,000 will render relatively quickly in ArcGIS. Inclusion of more than 500,000 polyfaces is not recommended.

f. Under More Options select Preserve Color.

g. Click “OK” to confirm and decimate the mesh.

Fig. 2: Select options for decimating the mesh.

[/wptabcontent]

[wptabtitle] Subsetting and Splitting Meshes[/wptabtitle] [wptabcontent]

(Skipping ahead a bit conceptually…) When you import your mesh data into ArcGIS each mesh is stored as a single multipatch. You don’t want to edit the shape of the multipatch in ArcGIS, only the placement (trust us on this). So any subsetting of the mesh needs to be performed before exporting from Rapidform (or other modeling software of your choice). Why subset or split a mesh?

a. Navigating in tight, enclosed spaces. You might want to be able to turn off the visibility of the back wall of a room or one half of a cistern to better visualize its interior.

b. Major sections of a mesh. If you have a scan of a building including several rooms or structures and you want to be able to visualize them individually, then they need to be made into discrete meshes.

[/wptabcontent]

[wptabtitle] Exporting[/wptabtitle] [wptabcontent]

a. Select the model you want to export from the menu tree on the left hand side of the screen.

b. Right-click and select “Export”. Select an appropriate file format (see step 2, below, for choices).

Fig. 3: Export via the menu tree.

4. Export Formats

a. Get a list of valid export formats by looking in the dropdown menu of the export dialog box.

Fig. 4: Valid export file formats.

b. Suggested formats for export are VRML (file extension .wrl), collada (.dae) and AutoDesk 3d Max (.3ds).

[/wptabcontent][wptabtitle] Advice on Textures and Color Data[/wptabtitle] [wptabcontent]

Modeling software manages color data in several ways. Color data might be recorded as UV coordinates referencing a separate texture file, as per vertex, per face or per wedge color information. Color data imported with scan data will typically default to storage as per vertex color. ArcGIS only recognizes color data stored explicitly in texture files, so if your color data is currently stored in another form you need to convert it.

[/wptabcontent]

[wptabtitle] Textures direct from Rapidform[/wptabtitle] [wptabcontent]

i. Select the Mesh mode from the main toolbar.

ii. Select Tools and Texture Tools and Convert Color to Texture.

Fig. 5: Conver Color to Texture

iii. After creating the texture, export the model as usual.

iv. Export the texture by going in the Main Menu to Texture Tools, then Export Texture to save the texture file. Store it in the same folder as the model.[/wptabcontent]

[wptabtitle] Color and Texture in Meshlab[/wptabtitle] [wptabcontent]Sometimes you want more tools for color editing. Sometimes ArcGIS doesn’t like the textures produced by Rapidform. For this reason, we suggest an alternative method for setting the texture data using Meshlab. Meshlab is open source, and can be found at meshlab.sourceforge.net.

i. From Rapidform export a .VRML file by right-clicking (in the model tree menu on the left land side of the screen) on the mesh you wish to export and selecting Export.

ii. In Meshlab, open a new empty project. Go to File and Import Mesh.

Fig. 6: Import the Mesh to Meshlab

iii. Select the VRML file you just created and hit Open.

iv. Transfer the color information from per vertex to per face. In the main menu go to Filters, then to Color Creation and Processing, then to Transfer Color: Vertex to Face. Hit Apply in the resulting pop-up menu.

Fig. 7: Transfer color data from the vertices to the faces of the mesh.

v. From the Main Menu go to Filters, then to Texture, then to Trivial Per-Triangle Parametrization.

Fig. 8: Create texture data.

In the pop-up menu, select 0 Quads per line, 1024 for the Texture Dimension, and 0 for Inter-Triangle border. Choose the Space Optimizing method. Click Apply.

n.b. If you get an error along the lines of “Inter-Triangle area is too much” your Texture Dimension is too small for the dataset. Increase the texture dimension to resolve the error.

Fig. 9: Set the texture data parameters.

vi. In the Main Menu go to Filters and Texture and Vertex Color to Texture. Accept the defaults for the name and size. Tick the boxes next to Assign texture and Fill Texture.

Fig. 10: Transfer color data to the texture dataset.

vii. In the Main Menu go to File and Export Mesh. Make sure to UNTICK the box next to Vertex Color. Otherwise ArcGIS gets confused! Make sure the texture file is present. Click OK to save.

Fig. 11: Export the mesh as collada (dae).[/wptabcontent]

[wptabtitle] Preparing a GIS to receive Mesh data[/wptabtitle] [wptabcontent]

Once you have created your mesh files and exported them to collada or something similar and explicitly assigned texture data (not to be confused with vertex color, face color or wedge color data), you are ready to import the data into ArcGIS. Assuming your data is not georeferenced, follow the method below. If your data is georeferenced, head over to our Photoscan to ArcGIS post, and follow the import method described there.

1. Preparing the geodatabase

a. Open ArcCatalog any way you choose. Create a new geodatabase by right clicking on the folder where you wish to create the geodatabase and selecting New and File Geodatabase. Only Geodatabases support the import of texture data, so don’t try and use a shapefile.

Fig. 14: Create a geodatabase in ArcGIS.

b. Create a multipatch feature class in the geodatabase.

c. Ensure that the X/Y domain covers the coordinates of any meshes you will be importing. View the Spatial Domain by right-clicking on the feature class and going to Properties and then to the Domain tab.

Fig. 15: Check the spatial domain of the new feature class.

d.If the spatial domain is not suitable, adjust the Environment settings by going to the Geoprocessing toolbar in the Main Menu. Scroll down to Geodatabase Advanced and adjust the Output XY Domain as needed. You can also adjust the Z Domain in this dialog box.

Fig. 16: Adjust the spatial domain in the environment settings.

[/wptabcontent]

[wptabtitle] Preparing the scene file. [/wptabtitle] [wptabcontent]

a. Open ArcScene and add base data such as a plan of the site, an air photo of the location, etc. The base data will allow you to control the location to which the model is imported. Add the empty multipatch feature class you just created.

Fig. 17: Add base data to a Scene.

b. Start editing either from the 3D editor toolbar or by right-clicking on the multipatch feature class in the Table of Contents and choosing Edit Features and Start Editing.

Fig. 18: Start editing in ArcScene. [/wptabcontent][wptabtitle] Importing the Scan data[/wptabtitle] [wptabcontent]

1. Import the vrml or collada file by selecting the Create Features Template for the multipatch and clicking on the base plan roughly in the location where you would like the mesh data to appear. Select the vrml or collada file from the Open File dialog box that appears. Wait while the file is converted.

Fig. 19: The vrml data is converted to multipatch on import.

2. You can now Move, Rotate, Scale the imported multipatch in ArcScene by selecting the feature using the Edit Placement tool and inputting values in the 3D Editing toolbar or by interactively dragging the multipatch feature.

Fig. 20: Select the multipatch feature to adjust its position and scale.

3. Once you are satisfied with the placement of the multipatch, you can add attribute data.

[/wptabcontent]

[wptabtitle] A note on rotation in Arcscene[/wptabtitle] [wptabcontent]

You can only rotate in the x-y plane (that is, around z-axis) in ArcScene. If you need to rotate your data around the x or y axis you need to do this in your modeling software before import. Bringing a .dxf of the polygon or point data you are trying to align the mesh with into your modeling software is probably the simplest way to get the alignment right. You may have to translate your .dxf to a local grid because most modeling software doesn’t like real world coordinates. Losing the real coordinates during this step doesn’t matter because you’re just using the polygon data to set orientation around the x and y axes. You’ll get the model in the correct real-world place when you import into ArcScene.

[/wptabcontent]

[wptabtitle] Re-exporting[/wptabtitle] [wptabcontent]

Fig. 21: The textured mesh data appears over the correct location on the base plan.

4. At this point it’s probably a good idea to re-export a collada model of your newly scaled and located mesh data. If not, every time you update the model you will have to go through the scaling and locating process again.

a. In ArcToolbox go to Conversion Tools> To Collada> Multipatch To Collada.

Fig. 22: Export Multipatch to Collada

b. Select the multipatch for export and the folder where you want the re-exported model to appear.

Fig. 23: Set parameters for export.

c. Check that the model has exported correctly by opening it in your modeling software.

n.b. You may have to reapply the textures at this point.

[/wptabcontent]

[wptabtitle] A note on features for attribute management [/wptabtitle] [wptabcontent]

It may be convenient to store attribute information in other related feature classes so that a single meshed model can have multiple, spatially discrete attributes. How you design your geodatabase will vary greatly dependent on project requirements.

Fig. 19: Additional related feature classes can be used to manage attribute data.

[/wptabcontent]

[wptabtitle] A note on just how much mesh data you can get into ArcScene.[/wptabtitle] [wptabcontent]

1. If you are using a file geodatabase, in theory the size of the geodatabase is unlimited and you can include all the mesh data you want.

2. In practice, individual meshes with more than 200,000 polygons have problems importing on an average desktop computer.

3. In practice, rendering becomes slow and jumpy with more than 200 MB of mesh data loaded into a single scene on an average desktop computer. The size and quality of your textures will also have an impact here. Compressed textures are probably a good plan.

4. In short, the limitation is on rendering and on what can be cached in an individual scene, rather than on storage in the geodatabase. Consider strategies including having low polygon count meshes for display in a general scene, with links to high polygon count meshes, which can be stored in the geodatabase but not normally rendered in the scene, which can be called up via links in html popup, the attribute table, or via another script.

[/wptabcontent]

[/wptabs]

For the .jpg, .tif, or .img photogrammetry formats, we recommend the free viewer ArcGIS Explorer Desktop. This free GIS application provides ways to explore and share GIS data.

dem_1972_1m.tif (File size – 71 mb)

Photogrammetric processing was performed on 10 historic vertical aerial photographs from 1972 to produce this digital elevation model (DEM) covering the monumental core and surrounding areas of Tiwanaku.
Ground sample distance – 0.5-m
Coverage – 330-ha
Coordinate system – Arbitrary, based on local coordinate system used by archaeologists.

Please note. This data is distributed under a Creative Commons 3.0 License (see http://creativecommons.org/licenses/by-nc/3.0/ for the full license). You are free to share and remix these data under the condition that you include attribution as provided here. You may not use the data or products in a commercial purpose without additional approvals. Please attach the following credit to all data and products developed there from:
Credit: Museum of Archeology and Anthropology, General Robotics, Automation, Sensing and Perception (GRASP) Lab (University of Pennsylvania) and Center for Advanced Spatial Technologies, (University of Arkansas)
Longer version: Data acquired, processed and distributed by the Center for Advanced Spatial Technologies staff and University of Pennsylvania.

The Center has been involved in a multi-year project in collaboration with Dr. Alexei Vranich at the University of Pennsylvania to scan and document the Pre-Incan site of Tiwanaku, Bolivia.  Read a short synopsis of the project at Tiwanaku Project Details and for full details on the entire survey, refer to Geophysics and Geomatics at Tiwanaku.

For the .jpg, .tif, or .img photogrammetry formats, we recommend the free viewer ArcGIS Explorer Desktop. This free GIS application provides ways to explore and share GIS data.

dem_1992_1m.tif (File size – 26 mb)

Photogrammetric processing was performed on two historic vertical aerial photographs from 1992 to produce this digital elevation model (DEM) covering the monumental core and surrounding areas of Tiwanaku.
Ground sample distance – 1-m
Coverage – 550-ha
Coordinate system – Arbitrary, based on local coordinate system used by archaeologists.

Please note. This data is distributed under a Creative Commons 3.0 License (see http://creativecommons.org/licenses/by-nc/3.0/ for the full license). You are free to share and remix these data under the condition that you include attribution as provided here. You may not use the data or products in a commercial purpose without additional approvals. Please attach the following credit to all data and products developed there from:
Credit: Museum of Archeology and Anthropology, General Robotics, Automation, Sensing and Perception (GRASP) Lab (University of Pennsylvania) and Center for Advanced Spatial Technologies, (University of Arkansas)
Longer version: Data acquired, processed and distributed by the Center for Advanced Spatial Technologies staff and University of Pennsylvania.

The Center has been involved in a multi-year project in collaboration with Dr. Alexei Vranich at the University of Pennsylvania to scan and document the Pre-Incan site of Tiwanaku, Bolivia. Read a short synopsis of the project at Tiwanaku Project Details and for full details on the entire survey, refer to Geophysics and Geomatics at Tiwanaku.

For the .jpg, .tif, or .img photogrammetry formats, we recommend the free viewer ArcGIS Explorer Desktop. This free GIS application provides ways to explore and share GIS data.

ortho_1992.tif (File size – 125 mb)

Photogrammetric processing was performed on two historic vertical aerial photographs from 1992 to produce this ortho mosaic covering the monumental core and surrounding areas of Tiwanaku.
Ground sample distance – 20.4-cm
Coverage – 710-ha
Coordinate system – Arbitrary, based on local coordinate system used by archaeologists.

Please note. This data is distributed under a Creative Commons 3.0 License (see http://creativecommons.org/licenses/by-nc/3.0/ for the full license). You are free to share and remix these data under the condition that you include attribution as provided here. You may not use the data or products in a commercial purpose without additional approvals. Please attach the following credit to all data and products developed there from:
Credit: Museum of Archeology and Anthropology, General Robotics, Automation, Sensing and Perception (GRASP) Lab (University of Pennsylvania) and Center for Advanced Spatial Technologies, (University of Arkansas)
Longer version:Data acquired, processed and distributed by the Center for Advanced Spatial Technologies staff and University of Pennsylvania.

The Center has been involved in a multi-year project in collaboration with Dr. Alexei Vranich at the University of Pennsylvania to scan and document the Pre-Incan site of Tiwanaku, Bolivia.  Read a short synopsis of the project at Tiwanaku Project Details and for full details on the entire survey, refer to Geophysics and Geomatics at Tiwanaku.

For the .jpg, .tif, or .img photogrammetry formats, we recommend the free viewer ArcGIS Explorer Desktop. This free GIS application provides ways to explore and share GIS data.

ortho_1972.tif (File size – 745 mb)

Photogrammetric processing was performed on 10 historic vertical aerial photographs from 1972 to produce this ortho mosaic covering the monumental core and surrounding areas of Tiwanaku.
Ground sample distance – 6.5-cm
Coverage – 360-ha
Coordinate system – Arbitrary, based on local coordinate system used by archaeologists.

Please note. This data is distributed under a Creative Commons 3.0 License (see http://creativecommons.org/licenses/by-nc/3.0/ for the full license). You are free to share and remix these data under the condition that you include attribution as provided here. You may not use the data or products in a commercial purpose without additional approvals. Please attach the following credit to all data and products developed there from:
Credit: Museum of Archeology and Anthropology, General Robotics, Automation, Sensing and Perception (GRASP) Lab (University of Pennsylvania) and Center for Advanced Spatial Technologies, (University of Arkansas)
Longer version: Data acquired, processed and distributed by the Center for Advanced Spatial Technologies staff and University of Pennsylvania.