geodesic.boson.dataset.Dataset.from_geoparquet#

static Dataset.from_geoparquet(name, url, feature_limit=10000, datetime_field='datetime', expose_partitions_as_layer=True, s3_region=None, s3_endpoint=None, credential=None, middleware={}, cache={}, tile_options={}, domain='*', category='*', type='*', **kwargs)[source]#

Creates a dataset from Hive-partitioned GeoParquet files in cloud storage.

Hive-partition GeoParquet is a particular convention typically used when writing data out from a parallel process (such as Tesseract or Apache Spark) or when the individual file sizes or row counts are too large. This provider indexes these partitions spatially to optimize query performance. Hive partitioned parquet is organized like this and we require this structure:

prefix/<root>.parquet

/key=value_1/<partition-00001>.parquet /key=value_2/<partition-00002>.parquet /… /key=value_m/<partition-n>.parquet

“root” and “partition-xxxxx” can be whatever provided they both have the parquet suffix. Any number oof key/value pairs are allowed in Hive Partitioned data. This can also point to a single parquet file.

Parameters:

  • name (str) – name of the Dataset to create

  • url (str) – the path to the prefix that contains the parquet partitions of interest. Format depends on the storage backend.

  • feature_limit (int) – the max number of features that this provider will allow returned by a single query.

  • datetime_field (str) – if the data is time enabled, this is the name of the datetime field. This is the name of a column in the parquet dataset that will be used for time filtering. Must be RFC3339 formatted in order to work.

  • expose_partitions_as_layer (bool) – this will create a collection/layer in this Dataset that simply has the partition bounding box and count of features within. Can be used as a simple heatmap

  • s3_region (str | None) – for S3 buckets, the region where the bucket is

  • s3_endpoint (str | None) – for S3 buckets, the endpoint to use (e.g. https://data.source.coop).

  • credential (str | None) – the name of the credential to access the data in cloud storage.

  • middleware (MiddlewareConfig | list) – configure any boson middleware to be applied to the new dataset.

  • cache (CacheConfig) – configure caching for this dataset

  • tile_options (TileOptions) – configure tile options for this dataset

  • domain (str) – domain of the resulting Dataset

  • category (str) – category of the resulting Dataset

  • type (str) – the type of the resulting Dataset

  • **kwargs – additional arguments that will be used to create the STAC collection, Dataset description Alias, etc.

Returns:

a new Dataset.

Return type:

Dataset

Examples

>>> ds = Dataset.from_geoparquet(
...          name="my-dataset",
...          url="s3://my-bucket/myprefix",
...          datetime_field="datetime",
...          credential="my-iam-user",
...          description="my dataset is the bomb"
...)
>>> ds.stage()
>>> # Staging is optional, but is a useful tool for validating configuration
>>> ds.save()
dataset:*:*:*:my-dataset
On this page

This Page