Oracle Big Data Spatial and Graph: Spatial Raster Analysis

In this topic, you install the Cartographic Projections library, load raster images to HDFS, and then create a basic mosaic from the images.

First,download the Cartographic Projections library and perform several related operations outlined in the next several steps:

1. Open a terminal window and execute the following two lines of code:

   sudo service bdsg stop
   sudo service bdsg status

Result: Oracle BigData Spatial and Graph Consoles are stopped.

2. Next, Launch Firefox and download this tarball (​http://download.osgeo.org/proj/proj-4.9.1.tar.gz) to ~/Downloads.

3. In the terminal window, untar the library source code by executing the following two lines of code:

   cd ~/Downloads
   tar -xzf proj-4.9.1.tar.gz
   

4. Configure, make, and install PROJ.4, by executing the following four lines of code:

   cd proj-4.9.1
   ./configure
   make
   sudo make install

   Result: libproj.so is now available at /usr/local/lib/.

5. Create a link to the libproj.so file in the spatial installation directory by executing the following command:

   sudo ln -s /usr/local/lib/libproj.so /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib/libproj.so

6. Provide read and execute permissions for the libproj.so library for all users by executing the following command:

   sudo chmod 755 /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib/libproj.so

7. Restart Tomcat and Spatial consoles by executing the following command.

   sudo service bdsg start

   Next, use the Raster Console to load raster image files into HDFS.

8. Open a new tab in the browser by clicking on the New Tab tool.

9. Then, select Spatial and Graph > Oracle Spatial Raster Console from the toolbar, as shown here:

   View Image

   

10. Select the Hadoop Raster Loader tab. Then, under the Input section, click the Add an input folder or file tool, as shown here:

    View Image

    

11. A. In the File System dialog, navigate to the /opt/shareddir/spatial/data/rasters directory.

    B. Then, using the CTRL key, multi-select the following three images (hawaii.tif, kahoolawe.tif, maui.tif), and click OK, as shown here:.

    View Image

    

    Result: The three images are added to the Input list, as shown here:

    View Image

    

12. Then, click the Load Images button, located just below the image file list.

    Result: When the images have been transfered to the HDFS system, a message similar to the following is displayed:

    View Image

    

13. As shown in the image above, click Return.

14. Next, select the Hadoop Raster Viewer tab, select the World Map map type, and then click on Refresh Footprints, as shown here:

    View Image

    

    Notes:

    • Wait until the image is refreshed.
    • Using the Slider, zoom in on the map so that only the United States is visible.
    • Then drag the map toward the Pacific Ocean, and zoom in on the location of the Hawaian lslands.

    You should see the following, with all three features drawn and highlighted:

    View Image

    

15. Click on each of the footprints individually. A thumbnail of the selected image appears, providing additional information about the image, as shown below:

    View Image

    

When you are done viewing additional information about the images, close the last thumbnail tab. Next, you learn how to create a basic mosaic.

16. First, as shown in the image below, click the Select Footprints tool at the upper left of the map. This action prepares the map for a mosaic area selection.

    View Image

    

17. Next, use the mouse to select all three footprints, as shown below:

    View Image

    

    Note: This action defines sub set operations on the rasters that are included within the selection.

18. Next, right-click on the map outside of the selection, and choose Generate Mosaic from the pop-up menu.

    Result: An HDFS Resource Manager form appears. In the form, coordinates, size and SRID values are generated automatically from the selected rasters.

19. In the form, you provide an output name for the mosaic. As highlighted in the image below, you first must click the elipses button to open a dialog for this purpose.

    View Image

    

20. In the output file name dialog, enter HawaiiMosaic as the name and click Ok.

21. Then, click the Create Mosaic button to start the mosaic and the subset operations generation.

    View Image

    

22. When the job is complete, the mosaic image is generated and displayed in a separate window, as shown here:

    View Image

    

    Notes: Use the window scroll bars to view more of the output.

    View Image

    

    Notes: Optionally, you can scroll down in the window and select the Download mosaic to save the mosaic locally. Or, if you want to generate a new mosaic with a different configuration, select the Go Back button.

23. When you are done viewing the mosaic, close the output window.

In this topic, you load a special raster image called a DEM, or Elevation Mode image, which supports a slope calculation for three-dimensional viewing. You will visualize this image by loadiong a custom jar into the image server.

NOTE: You must succesfully complete this topic in order to perform the next topic.

Follow these steps:

1. In the browser, open a new tab and go back to Spatial and Graph > Oracle Spatial Raster Console.

2. In the Image Server window, select the Hadoop Raster Loader tab. As before, navigate to and select the /opt/shareddir/spatial/data/rasters/NapaDEM.tif image as shown here:

   View Image

   

3. Click OK.

4. Then, click the Load images button, and wait for the image to be loaded.

   Result: When the image has been transfered to the HDFS system, the following message appears:

   View Image

   

5. Click Return.

6. Select the Hadoop Raster Viewer tab, select World Map as the Map Type, and then and click on the Refresh Footprints button as before.

   Result: The message "Features Loaded: 4" appears next to the Refresh Footprints button, and the image is refreshed.

   View Image

   

7. Using the mouse, relocate the map to center on California, and zoom in on the Napa Valley area, as shown below.

   View Image

   

   Note: As highlighted in the image above, the features for the NapaDEM image have been loaded for the the map. The footprint section of the map that is associated with these features is also highlighted.

   Next, you will create a mosaic of this footprint in using the same technique that you learned in the previous topic.

8. First, click the Select Footprints tool at the upper left of the map.

   View Image

   

9. Now, Use the mouse to select the entire footprint area. Right-click on the map outside of the selection to display a pop-up menu, and select Generate Mosaic as shown below:

   View Image

   

   Result: A Mosaic form appears, with coordinates, size and SRID values automatically generated, as you observed previously.

10. In the form, specify the Output folder value by using the elipses button. Enter slope as the file Mosaic file name.

11. Still in the form, select Float 32 Bits as the pixel type (this is a 32 bit image), as shown below:

    View Image

    

12. Next, click the Add process class button.

13. Then, provide the name of the class that will perform the slope calculation, which is: oracle.spatial.hadoop.imageprocessor.process.ImageSlope

    Result: The lower portion of the form should now look like this:

    View Image

    

    Note: The Full Name Class is the fully class qualified name, including the package name.

14. Finally, click the Advanced configuration... link, and provide the algebra operation as shown below, by performing the following:

    • Select the localif function.
    • Click on Add Operation
    • Specify the following parameter values for localif as shown below:
      • Operators = Greater Than
      • Operand = 2500
      • New value = 3900

    Result: The Operations section should look like this:

    View Image

    

    • As shown above, click OK to save the configuration.
15. Click the Create Mosaic button to start the mosaic generation process.

    Result: As before, while the job is running you can view the running applications page by going to http://localhost:8088/cluster/apps in your browser. Click on the Running link as shown below to refresh the page.

    Result: When the job is complete,the mosaic image is generated. However, it is not fully visible in the Image Server viewer, as shown below:

    View Image

    

16. Close the image viewer window.

    Notes:

    • In order to fully visualize the image, you could download the image using the Download mosaic button. Then, you can copy to a Windows machine and open the file using Windows Photo Viewer. This is an optional step.
    • The following screenshot shows how this image looks using Windows Photo Viewer, including the slope aspects:

    View Image

    

In this topic, you use the same the raster image from the previous topic. Here, you will use an external jar file with the image, as an example of how the spatial raster framework can be extended. In addition, you learn how to perform new raster analysis calculation, called Hillshade.

Follow these steps:

1. In the Georaster Tools window, select the Administrator tab. Note: Ignore the "Save the changes permanetly" message.

2. Scroll down to the "Upload user jars for mosaic processing" section Then, click the Browse button, as shown below:

   View Image

   

3. In the File Upload dialog, navigate to the /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/jlib directory and select the external-analysis.jar file. Finally, click Open, as shown below:

   View Image

   

4. Then, click Upload to add the custom jar to the imageserver.

   View Image

   

   Result: When the jar successfully uploads, a message appears that states the new jar has been added.

5. Then, click Return to Console.

6. Now, select the Hadoop Raster Viewer tab, click the Remove Selection tool, and click Refresh Footprints.

7. Once again, click the Select and crop coordinates of Footprints tool at the upper left of the map.

   View Image

   

8. Use the mouse to select the entire footprint area, right-click on the map outside of the selection, and choose Generate Mosaic from the menu, as shown below:

   View Image

   

   Result: As before, a HDFS Resource Manager form appears, with coordinates, size and SRID values automatically generated.

9. In the form, use the elipses button to specify hillshade as the Mosaic file name.

10. Still in the form, select Float 32 Bits as the pixel type, as you did previously.

    View Image

    

11. Next, click the Add process class button and enter oracle.spatial.raster.HillShade as the Full Name Class.

    Result: The lower portion of the form should now look like this:

    View Image

    

12. Click Create Mosaic as shown above to generate mosaic.

    Note: Wait for the result, as the image may take a few moments to appear.

    Result: The hillshade mosaic image should look like this:

    View Image

    

13. Close the image viewer window and then close the browser.

In this exercise, you load a set of raster images with different resolutions, data types and SRID´s, into HDFS.

Theses images include the four that you have worked with in previous exercises:

View Image

Follow these steps:

1. Using a terminal window, execute the following long line of code, which includes several required parameters, in order to load the images:

   hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageloader.jar -files /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/raster/hawaii.tif,/opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/raster/kahoolawe.tif,/opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/raster/maui.tif,/opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/raster/NapaDEM.tif  -out exercise0 -overlap 10 -thumbnail /opt/shareddir/spatial -gdal  /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib  -gdalData /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/data

2. Once the job is finished, execute the following command to verify that this the raster images loaded correctly.

   hdfs dfs -ls /user/oracle/exercise0/opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/raster

   You should see a list of images files, as shown here:

   View Image

   

3. Leave the terminal window open.

In this exercise, you process the three Hawaii raster images that you just loaded by extracting them and creating a mosaic with them.

Before running the job, view two .xml files that are used in the raster processing. Both files are located in the /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml directory. In our example, we use gedit to open the files.

First, the index.xml file is opened:

View Image

Notes: The raster directory set in HDFS includes “exercise0” as part of the path for each raster in the catalog, since this is the directory used for output in the loader process.

Second, the testFS.xml file is opened:

View Image

Notes: As shown, the xml code defines a mosaic that includes all three rasters. The specified coordinates in the transform element includes 3 bands, an SRID of 26904, and GTIFF format. In addition, the directory type element specifies that the mosaic file output will be located in NFS.

To run the raster image process job, perform the following:

1. In the terminal window, execute the following long command to process the catalog and mosaic.

2. hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageprocessor.jar -catalog /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml/input.xml -config /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml/testFS.xml -gdal /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib  -gdalData /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/data

   Result: When the job is finished, all the rasters in the catalog will be combined in a single raster output based on the configuration xml details.

3. Launch the broswer and open a new tab. Then open the /opt/shareddir/spatial/hawaiimosaic.tif file. Click OK to accept the default viewer, as shown below:

   View Image

   

   Result: The mosaic should look like this:

   View Image

   

   Note: All the three of the catalog rasters are together in this mosaic.

4. When done viewing the image, close the window.

This exercise uses the Hawaii raster images loaded previously. Here, you extract a subset of the rasters and transform the pixels using a raster algebra function called localnot. (This function was also used in one of the exercises in the Raster Console topic.) In this case, localnot inverts every bit in the pixels before creating a mosaic.

As with the previous exercise, two .xml files are used in the processing job. And again, both files are located in the /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml directory.

The first file is the same index.xml file that was used previously, shown here:.

View Image

The second file used for this processing job is testFSPartial.xml, shown below:

View Image

Notes:

- In this case, the xml code defines a mosaic that includes all three rasters with the same coordinates in the transform elements (3 bands, an SRID of 26904, and GTIFF format) as used in the previous exercise. And again, the directory type element specifies that the mosaic file output will be located in NFS.

- However, in this case the transform coordinates are changed from the last exercise. Now the mosaic has a minor pixelWidth and pixelHeight, which will cut off the last island on the right (the hawaii.tif raster), and you will visualize this on the mosaic when the output is processed.

- In addition, there is a new element in this xml, <operations>, where localnot operation is defined. This section of the xml is highlighted in the screenshot above. As mentioned previously, this operation inverts the bits in every pixel and is part of the supported raster algebra operations included in the framework.

To run this raster image process job, perform the following:

1. Back in the terminal window, execute the following long command to process the catalog and mosaic.

   hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageprocessor.jar -catalog /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml/input.xml -config /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml/testFSPartial.xml -gdal /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib -gdalData /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/data

2. After the job finishes, open a new tab in the browser and open /opt/shareddir/spatial/hawaiimosaic.tif. The mosaic file is shown below:

   View Image

   

   Notes: In this example, the last island on the right is cut, this is a result of the transform coordinates set in the mosaic configuration. This is an example of subset operation. Also, you may notice the change in the pixels colors, which is a result of applying the localnot operation.

3. When done viewing the image, close the window.

In this exercise, you process the Napa Valley Elevation Model loaded in HDFS previously. In this case, the localif function is applied.

As with the previous exercise, two .xml files are used in the processing job. And again, both files are located in the /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml directory.

The first file is inputDEM.xml, which is shown here:.

View Image

Notes: Again, the raster directory contains “exercise0” as part of the path for the raster in the catalog. As in previous examples, this file will be used when executing the job to set the catalog using the –catalog option.

The second file used for this processing job is testSlopeFS.xml, shown below:

View Image

Notes: In addition to options that have been previously specified, this xml file uses the <process> element, which indicates the ImageSlope class. When executed as part of the process, the mosaic output will show the slope according to its neighboring pixels. Also, the <operations> element sets the localif function. This operation updates the value of every pixel where its original elevation is greater than 2500.

To run this raster image process job, perform the following:

1. Back in the terminal window, execute the following long command to process the catalog and mosaic.

   hadoop jar /opt/oracle/oracle-spatial-graph/spatial/raster/jlib/hadoop-imageprocessor.jar -catalog /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml/inputDEM.xml -config /opt/oracle/oracle-spatial-graph/spatial/raster/Raster-HOL/data/xml/testSlopeFS.xml -gdal /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/lib -gdalData /opt/oracle/oracle-spatial-graph/spatial/raster/gdal/data

   Notes: After the job finishes, the /opt/shareddir/spatial directory contains the mosaic, as shown here:

   View Image

   

   However, recall that the output raster with the slope cannot be visualized in Linux environment with the standard image tools. Therefore, the mosaic file is shown below in a Windows environment using Windows Photo Viewer:

   View Image

   

2. In the Big Data VM, close the browser and all open terminal windows.