Oracle Big Data Spatial and Graph: Spatial Vector Analysis

In this topic, you create an index on the file /user/oracle/HOL/tweets.json.

Follow these steps:

1. Launch the browser, and select Spatial and Graph > Oracle Spatial Vector Console from the toolbar, as shown here:

   View Image

   

2. In the Spatial Hadoop Vector Console, click the Spatial Index tab, and then specify the following values:
   • Index name: tweetsJanuaryIndex
   • Path of data to the index: hdfs://bigdatalite.localdomain:8020/user/oracle/HOL/tweets.json
   • New index path: hdfs://bigdatalite.localdomain:8020/user/oracle/HOL/tweetsIndex1
   • SRID (of the geometries used to build the index): 8307
   • Tolerance (to accommodate rounding errors): 0.5
   • Input Format class (used to read the input data): oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat
   • Record Info Provider class (provides spatial information): oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider

   The completed New Index form should look like this:

   View Image

   

3. As shown in the image above, click Create.

4. When the job is submitted, open a new Firefox tab and then go to http://localhost:8088/cluster/apps.

In the Hadoop - All Applications tab, click the FINISHED link, as shown here:

View Image



Note: When the job completes successfully, the log should show a State of FINISHED, and a FinalStatus of SUCCEEDED, as shown below.

View Image



5. Close the Hadoop - FINISHED Applications tab.

Using the spatial index that you completed in the previous topic, you categorize the tweets in the file /user/oracle/HOL/tweets.json by Countries, States, and Provinces in order to determin how many tweets have been sent by country and by State or Province.

1. In the Spatial Hadoop Vector Console, Select Categorization > Run Job.
2. In the New Job form, select tweetsJanuaryIndex from the Index option, as shown below:

   View Image

   

3. Next, click the Select Templates button, and choose World Countries as hierarchy 1, and World State Provinces as hierarchy 2. Then click Save, as shown below:

   View Image

   

4. Finally, complete the New Job form, as shown below:
   • Output path: hdfs://bigdatalite.localdomain:8020/user/oracle/HOL/catOutput
   • Name: Tweets January
   • Click Create

5. View Image

   

6. Again, open http://localhost:8088/cluster/apps in a new Firefox tab. When the job is completes, you should see the Hierarchical job having finished successfully, as shown here:

   View Image

   

7. Toggle back to the Spatial Hadoop Vector Console tab and select Categorization > View Results.

8. First, in the Templates list, click on the World Countries. A Results category appears below the Templates list.

9. Then, click on Tweets January result. A color-coded map of the world countries appears. Note: You can zoom on the map by double-clicking the image.

10. Move the cursor over the United States to view the number of tweets in the US.

    View Image

    

11. Now, click the country United States.

    Mouse over any state to view the number of tweets per state. In this example, the tweet count for Alaska is shown:

    View Image

    

    NOTES: You can refersh the map and view other locations by selecting Templates > World Countries and Results > Tweets January on the left. When you are done viewing tweet results in the map, move on to the next exercise.

In this topic, you run a binning analysis job on the tweets, using the same index that you created previously.

Follow these steps:

1. In the Spatial Hadoop Vector Console tab, select Binning-> Run Job.

2. Then, in the New Job form, select the With Index option and tweetsJanuaryIndex

3. Change the binning grid minimum bounding rectangle (MBR) to the following values:
   • Min. X = -175
   • Max. X = 175
   • Min. Y = -75
   • Max. Y = 75

4. Then, click the Display MBR button (as shown below):

   View Image

   

   Result: The following map appears in a new window.:

   View Image

   

5. Back in the New Job form, set the Binning Shape to Hexagon, and the cell width of each hexagon is set to 5.

   View Image

   

6. Finally, use the following values for these remaining options:

   • Leave the Thematic attribute option value as count. This value means that each cell in the grid will display the number of tweets associated with that cell.
   • Output path: hdfs://bigdatalite.localdomain:8020/user/oracle/HOL/binningOutput
   • Result Name: Tweets January

   The completed New Job form should look like this:

   View Image

   

7. Now, click Preview to display a preview of the grid with empty hexagonal bins in the map window, as shown below:

   View Image

   

8. Finally, close the preview map and click Create in the New Job form.

   NOTE: While the binning job is running, toggle back to the Applications tab in the browser to view the job log. Click the FINISHED link to refresh the log. Wait until log shows that the job completed successfully.

9. When the binning job is finshed, toggle back to the Spatial Hadoop Vector Console tab and select Binning > View Results to display the binning map.

10. In the options bar to the left of the map, select World Map as the Backround option, and click the Tweets January link under the Results heading.

    Result: You may view the number of tweets in any cell moving the mouse cursor on it. As before, you can double-click on the map to zoom in.

    View Image

    

    NOTES: The darker the color, the larger the number of tweets in that bin. When you are done viewing tweet results in the map, move on to the next exercise.

In this topic, you run a clustering analysis job on the tweets. The K–means clustering method is used and the number of clusters is 2.

NOTE: K-means is a popular method for clustering analysis in data mining. For more information, see: https://en.wikipedia.org/wiki/K-means_clustering.

Follow these steps:

1. In the Spatial Viewer tab, select Clustering > Run Job.
2. Then, in the New Job form, specify the following values:
• Path of data: hdfs://bigdatalite.localdomain:8020/user/oracle/HOL/tweets.json
• SRID: 8307
• Tolerance: 0.5
• Input Format class: oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat
• Record Info Provider class: oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider
• Output path: hdfs://bigdatalite.localdomain:8020/user/oracle/HOL/clusteringOutput
• Number of clusters: 2
• Result Name: Tweets January

The New Job form should look like this:

View Image



3. Click Create.

4. While the clustering job is running, toggle back to the Applications tab in the browser to view the job log. Click the FINISHED link to refresh the log. Wait until the log shows that the job has completed successfully.

5. When the job is finshed, toggle back to the console tab and select Clustering > View Results to display the clustering map.

6. In the options bar to the left of the map, select World Map as the Background and click the Tweets January link under the Results heading. Then, enable the Show clusters boundaries option.

   Notes: The Tweets January cluster result may take a moment to appear in the viewer. Click the refresh tool () next to the Results heading until the list is updated.

   Result: The map shows clusters centers and boundaries. As before, double click the image to zoom in.

   View Image

   

7. Mouse over a cluster to display the number of tweets inside that cluster. In this example, the East cluster has 2534 members, and the West cluster has 4169 members. Note: Your values may be slightly different.

   View Image

   

8. When you are done, close the browser.

In this topic, you create a spatial index using a job named oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing.

This job requires a variety of agruments to define the parameters for the index creation process. The parameters are similar to those that you used in the Vector Console form, when creating the vector index for the previous exercise.

The job:

• Uses the file /user/oracle/HOL/tweets.json as input data for the index
• Creates an index named indexGeoJSON
• Generates metadata for the index in a specified directory

To generate the index, follow these steps:

1. Using the terminal window, execute the following long line of code to generate the index:

   hadoop jar $API_LIB_DIR/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.SpatialIndexing -libjars $HADOOP_LIB_JARS input=/user/oracle/HOL/tweets.json output=/user/oracle/HOL/indexGeoJSON inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider srid=8307 geodetic=true tolerance=0.5 indexName=indexGeoJSON indexMetadataDir=/user/oracle/indexMetadata overwriteIndexMetadata=true

   Once the job is finished, the message "INFO: Job Succeeded" appears in the output.

In this topic, you run a job that categorizes tweets by countries and regions inside the eurozone.

Since the API doesn’t provide the needed country and region data sets, the job will use two JSON format files: eurozone_countries.json and eurozone_provinces.json.

In addition, a HierarchyInfo class is called by the job. This class describes the hierarchies that are used to categorize the tweets data. The first hierarchy is the eurozone_countries and the second hierarchy is the eurozone_provinces.

An image of the class implementation is shown here:

View Image

Note: The class is contained in a jar file named eurohierarchyinfo.jar.

Follow these steps:

1. Using the open terminal window, execute the following two export commands to make eurohierarchyinfo.jar available for Hadoop:

   export HADOOP_CLASSPATH=/opt/oracle/oracle-spatial-graph/spatial/vector/HOL/jlib/eurohierarchyinfo.jar:$HADOOP_CLASSPATH
   export HADOOP_LIB_JARS=/opt/oracle/oracle-spatial-graph/spatial/vector/HOL/jlib/eurohierarchyinfo.jar,$HADOOP_LIB_JARS

2. Now run the categorization job by executing the following command in the terminal window:

   hadoop jar $API_LIB_DIR/sdohadoop-vector.jar oracle.spatial.hadoop.vector.mapred.job.Categorization -libjars $HADOOP_LIB_JARS spatialOperation=IsInside input=/user/oracle/HOL/tweets.json output=/user/oracle/HOL/euroZoneCatOutput inputFormat=oracle.spatial.hadoop.vector.geojson.mapred.GeoJsonInputFormat recordInfoProvider=oracle.spatial.hadoop.vector.geojson.GeoJsonRecordInfoProvider srid=8307 geodetic=true tolerance=0.5 hierarchyInfo=hol.EuroHierarchyInfo hierarchyIndex=/user/oracle/HOL/hierarchyIndex hierarchyDataPaths=file:///opt/oracle/oracle-spatial-graph/spatial/vector/HOL/data/eurozone_countries.json,file:///opt/oracle/oracle-spatial-graph/spatial/vector/HOL/data/eurozone_provinces.json

   Note: Once the job completes, the message "INFO: Job Succeeded" appears in the output. In addition, the result is saved in the HDFS folder /user/oracle/HOL/euroZoneCatOutput.

3. Using the terminal window, execute the following command to copy the output locally:

   sudo hadoop fs -get /user/oracle/HOL/euroZoneCatOutput/*.json

4. Then, execute the ls command to view a listing of the tweet count output files:

   ls

   View Image

   

5. Execute the following command to view some of the data for the eurozone contries count output:

   more eurozone_countries_count.json

   A portion of the count output is shown here:

   View Image

   

   Notes:

   Each record id refers to the record in the eurozone_countries.json file. For example:

   • The count result highlighted in the image above ({"id":"FRA","result":62}) refers to the following record in the file eurozone_countries.json:
   • {"type":"Feature","_id":"FRA","geometry":{"type":"Polygon","coordinates":[[1.44136,42.60366,1.47851,42.65168,…,1.44136,42.60366]]},"properties":{"Continent":"EU","Name":"France","Alt_Region":"EMEA","Country Code":"FRA"},"label_box":[-1.12061,45.13915,6.02255,49.19591]}

In this case, we see that 62 tweets have been sent from France.

6. Close any open terminal windows.