Working with Multiple Data Sources in your Environment of Choice - ArcGIS Online

Problem

You want to use Boson to bring multiple datasets from different sources, together into one place.

Solution

Boson makes bringing disparate datasets together very easy. In this demo, we will bring together data from flat files, STAC collections and ArcGIS online. Once Boson pulls the datasets together in a uniform way, we can serve them out in a variety of ways, depending on our needs.

Setup

First, let’s setup the environment and create a project, if you don’t already have one set up. You can run through the basic setup in Geodesic Basics. Then set the active project.

import geodesic
geodesic.set_active_project('docs-example')

Load the datasets

In other cookbooks, we have shown how to load a variety of datasets. Here, we will make use of a few of them. In particular, we will load one dataset from a STAC collection, one from a cloud bucket, and one from an ArcGIS online service.

Load a dataset from a STAC collection

First, let’s load the STAC Collection. This is Sentinel 2 L2A raster data. You can examine the cookbook in more detail in the Adding a STAC Collection cookbook, but we really only need a few lines of code:

stac_link = "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs"
ds = geodesic.Dataset.from_stac_collection('sentinel-2',stac_link)
ds.save()

Load a dataset from a cloud bucket

Next, let’s load a dataset from a cloud bucket. Again, you can examine the cookbook in more detail in the Adding a GeoJSON Dataset cookbook, but we can load the dataset like so:

ds = geodesic.Dataset.from_tabular_data(
    name='uscb-pop-centers',
    url='gs://geodesic-public-data/CenPop2020_Mean_CO.geojson',
    index_data=True,
    crs='EPSG:4326',
    )
ds.save()

Load a dataset from an ArcGIS online service

Finally, let’s load a dataset from an ArcGIS online service. Again, you can examine the cookbook in more detail in the Adding an ArcGIS polygon_features Layer Dataset cookbook. Here is the relevant code:

url = "https://geodata.epa.gov/arcgis/rest/services/OEI/FRS_INTERESTS/MapServer/7"
ds = geodesic.Dataset.from_arcgis_layer(name = "epa-frs-locations", url=url)
ds.save()

It is worth noting that we don’t strictly need to pull this dataset into Boson if our intention is to add it to ArcGIS online, since it is already an ArcGIS online service. However, the point of this cookbook is to show that, once we have connected to a dataset with Boson, it does not matter what the source was. We now have a uniform way to interact with the data, and we can serve it out in a variety of ways, including but not limited to ArcGIS Online.

Bringing the datasets together

Now that our disparate datasets are saved in this project in a uniform representation, we can serve them out in a variety of ways. Here, we will serve them out to ArcGIS online.

First, let’s retrieve the datasets we just saved. We can do this by using the geodesic.entanglement.dataset.get_datasets() method. We can use the search parameter to query for the particular datasets we want. If we know the name of the dataset, we can use the geodesic.entanglement.dataset.get_dataset() method.

from geodesic.entanglement.dataset import get_datasets, get_dataset

datasets = get_datasets()
datasets

sentinel2_data = get_dataset('sentinel-2')
uscb_data = get_dataset('uscb-pop-centers')
epa_data = get_dataset('epa-frs-locations')

Now, we can create share links to serve the data.

Serving the Datasets to ArcGIS Online

We can serve the datasets out in a variety of ways. Let’s start by simply creating vector tile services for each dataset, and then we will serve them to an ArcGIS Online map. We will serve the Sentinel 2 data as a vector tile service, which will display the footprints of the imagery. Because Sentinel 2 data is large and contains 10 bands, we will first have to create a view of the dataset that is appropriate for serving as an feature service. Views allow us to apply a persistent filter to a dataset. Here, we will create a view that only includes the red band and a bounding box around Pennsylvania, USA for a 20-day period in January 2020.

import datetime

bbox = [-80.463867, 39.707187, -75.102539, 41.983994]
start_time = datetime.datetime(2020, 1, 1)
end_time = start_time + datetime.timedelta(days=20)

asset_bands = [{'asset': 'B04', 'bands': [0]}]

sentinel2_view = sentinel2_data.view(
    name='sentinel2-pa',
    bbox=bbox,
    datetime=[start_time, end_time],
    asset_bands=asset_bands,
)

sentinel2_view.save()

Now, we can create share links to serve the data. First, we create a share token. You can set an expiration time, in seconds, for the token. Here, we will set the expiration time to 3600 seconds (1 hour). Leaving the expiration time blank will create a token that never expires.

sentinel2_token = sentinel2_view.share_as_arcgis_service(3600)

Then, we can retrieve the URL for the service. Here, we will use the get_vector_tile_service_url() method to get the URL for the vector tile service.

Note

Alternatively, you can share features in ArcGIS with the methods get_feature_layer_url(), and get_feature_service_url().

Furthermore, Geodesic supports serving the data as an image service with the get_image_service_url() method. This will be covered in another cookbook as it requires additional steps.

sentinel2_url = sentinel2_token.get_vector_tile_service_url()

Next, we will create vector tile feature services for the other two datasets.

uscb_token = uscb_data.share_as_arcgis_service()
uscb_url = uscb_token.get_vector_tile_service_url()

epa_token = epa_data.share_as_arcgis_service()
epa_url = epa_token.get_vector_tile_service_url()

Finally, we can serve the datasets to an ArcGIS Online map.

Adding the Datasets to an ArcGIS Online Map

You can add the vector tile services to your ArcGIS Online account by using the urls we just created. First, add a new item.

Then, select “URL”.

Paste the URL for the vector tile service into the URL field.

print(sentinel2_url)

# Example output:
https://api.geodesic.seerai.space/ted/api/v1/share/1b0a6922868f493139106baf7cb2c85ec3b5e2343f9bb3798a25dbc5874c4730/rest/services/sentinel2-pa/VectorTileServer

Edit the title, folder and other options if desired, and click “Save”.

Now your vector tile service is available in your ArcGIS Online account as a tile layer.

You can add the other datasets in the same way, or you can open the Map Viewer and add them directly to a map. Here, we will open the Map Viewer to view the Sentinel 2 imagery footprints. By default, the Sentinel 2 imagery footprints will be opaque, as shown below.

You can edit the layer using ArcGIS online tools, which will make the footprints visible.

And we will add the other two datasets using the “Add layer from URL” option in the “Layers” pane.

And there you have it. Using Geodesic, we’ve easily assembled three datasets from three different sources and served them out to ArcGIS Online in a uniform way.