I/O¶
Read and write to and from geospatial file formats.
geoarrow.rust.io ¶
ObjectStore ¶
ParquetDataset ¶
crs ¶
crs(column_name: str | None = None) -> CRS
Access the CRS of this file.
Parameters:
-
column_name
(str | None
, default:None
) –The geometry column name. If there is more than one geometry column in the file, you must specify which you want to read. Defaults to None.
Returns:
-
CRS
–CRS
read ¶
read(
*,
batch_size: int | None = None,
limit: int | None = None,
offset: int | None = None,
bbox: Sequence[IntFloat] | None = None,
bbox_paths: BboxPaths | None = None
) -> Table
Perform a sync read with the given options
Parameters:
-
batch_size
(int | None
, default:None
) –description. Defaults to None.
-
limit
(int | None
, default:None
) –description. Defaults to None.
-
offset
(int | None
, default:None
) –description. Defaults to None.
-
bbox
(Sequence[IntFloat] | None
, default:None
) –description. Defaults to None.
-
bbox_paths
(BboxPaths | None
, default:None
) –description. Defaults to None.
Returns:
-
Table
–description
read_async
async
¶
read_async(
*,
batch_size: int | None = None,
limit: int | None = None,
offset: int | None = None,
bbox: Sequence[IntFloat] | None = None,
bbox_paths: BboxPaths | None = None
) -> Table
Perform an async read with the given options
Parameters:
-
batch_size
(int | None
, default:None
) –description. Defaults to None.
-
limit
(int | None
, default:None
) –description. Defaults to None.
-
offset
(int | None
, default:None
) –description. Defaults to None.
-
bbox
(Sequence[IntFloat] | None
, default:None
) –description. Defaults to None.
-
bbox_paths
(BboxPaths | None
, default:None
) –description. Defaults to None.
Returns:
-
Table
–description
ParquetFile ¶
crs ¶
crs(column_name: str | None = None) -> CRS
Access the CRS of this file.
Parameters:
-
column_name
(str | None
, default:None
) –The geometry column name. If there is more than one geometry column in the file, you must specify which you want to read. Defaults to None.
Returns:
-
CRS
–CRS
file_bbox ¶
Access the bounding box of the given column for the entire file
If no column name is passed, retrieves the bbox from the primary geometry column.
An Err will be returned if the column name does not exist in the dataset None will be returned if the metadata does not contain bounding box information.
read ¶
read(
*,
batch_size: int | None = None,
limit: int | None = None,
offset: int | None = None,
bbox: Sequence[IntFloat] | None = None,
bbox_paths: BboxPaths | None = None
) -> Table
Perform a sync read with the given options
Parameters:
-
batch_size
(int | None
, default:None
) –description. Defaults to None.
-
limit
(int | None
, default:None
) –description. Defaults to None.
-
offset
(int | None
, default:None
) –description. Defaults to None.
-
bbox
(Sequence[IntFloat] | None
, default:None
) –description. Defaults to None.
-
bbox_paths
(BboxPaths | None
, default:None
) –description. Defaults to None.
Returns:
-
Table
–description
read_async
async
¶
read_async(
*,
batch_size: int | None = None,
limit: int | None = None,
offset: int | None = None,
bbox: Sequence[IntFloat] | None = None,
bbox_paths: BboxPaths | None = None
) -> Table
Perform an async read with the given options
Parameters:
-
batch_size
(int | None
, default:None
) –description. Defaults to None.
-
limit
(int | None
, default:None
) –description. Defaults to None.
-
offset
(int | None
, default:None
) –description. Defaults to None.
-
bbox
(Sequence[IntFloat] | None
, default:None
) –description. Defaults to None.
-
bbox_paths
(BboxPaths | None
, default:None
) –description. Defaults to None.
Returns:
-
Table
–description
row_group_bounds ¶
Get the bounds of a single row group.
Parameters:
-
row_group_idx
(int
) –The row group index.
-
bbox_paths
(BboxPaths | None
, default:None
) –For files written with spatial partitioning, you don't need to pass in these column names, as they'll be specified in the metadata Defaults to None.
Returns:
row_groups_bounds ¶
row_groups_bounds(bbox_paths: BboxPaths | None = None) -> GeometryArray
Get the bounds of all row groups.
As of GeoParquet 1.1 you won't need to pass in these column names, as they'll be specified in the metadata.
Parameters:
-
bbox_paths
(BboxPaths | None
, default:None
) –For files written with spatial partitioning, you don't need to pass in these column names, as they'll be specified in the metadata Defaults to None.
Returns:
-
GeometryArray
–A geoarrow "box" array with bounds of all row groups.
ParquetWriter ¶
Writer interface for a single Parquet file.
This allows you to write Parquet files that are larger than memory.
close ¶
close() -> None
Close this file.
The recommended use of this class is as a context manager, which will close the file automatically.
write_batch ¶
write_batch(batch: ArrowArrayExportable) -> None
Write a single RecordBatch to the Parquet file
write_table ¶
write_table(table: ArrowArrayExportable | ArrowStreamExportable) -> None
Write a table or stream of batches to the Parquet file
This accepts an Arrow RecordBatch, Table, or RecordBatchReader. If a RecordBatchReader is passed, only one batch at a time will be materialized in memory.
Parameters:
-
table
(ArrowArrayExportable | ArrowStreamExportable
) –description
read_csv ¶
read_csv(
file: str | Path | BinaryIO,
geometry_column_name: str,
*,
batch_size: int = 65536
) -> Table
Read a CSV file from a path on disk into a Table.
Parameters:
-
file
(str | Path | BinaryIO
) –the path to the file or a Python file object in binary read mode.
-
geometry_column_name
(str
) –the name of the geometry column within the CSV.
-
batch_size
(int
, default:65536
) –the number of rows to include in each internal batch of the table.
Returns:
-
Table
–Table from CSV file.
read_flatgeobuf ¶
read_flatgeobuf(
file: Union[str, Path, BinaryIO],
*,
fs: Optional[ObjectStore] = None,
batch_size: int = 65536,
bbox: Tuple[float, float, float, float] | None = None
) -> Table
Read a FlatGeobuf file from a path on disk or a remote location into an Arrow Table.
Example:
Reading from a local path:
from geoarrow.rust.core import read_flatgeobuf
table = read_flatgeobuf("path/to/file.fgb")
Reading from a Python file object:
from geoarrow.rust.core import read_flatgeobuf
with open("path/to/file.fgb", "rb") as file:
table = read_flatgeobuf(file)
Reading from an HTTP(S) url:
from geoarrow.rust.core import read_flatgeobuf
url = "http://flatgeobuf.org/test/data/UScounties.fgb"
table = read_flatgeobuf(url)
Reading from a remote file on an S3 bucket.
from geoarrow.rust.core import ObjectStore, read_flatgeobuf
options = {
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_region": "..."
}
fs = ObjectStore('s3://bucket', options=options)
table = read_flatgeobuf("path/in/bucket.fgb", fs=fs)
Parameters:
-
file
(Union[str, Path, BinaryIO]
) –the path to the file or a Python file object in binary read mode.
Other Parameters:
-
fs
(Optional[ObjectStore]
) –an ObjectStore instance for this url. This is required only if the file is at a remote location.
-
batch_size
(int
) –the number of rows to include in each internal batch of the table.
-
bbox
(Tuple[float, float, float, float] | None
) –A spatial filter for reading rows, of the format (minx, miny, maxx, maxy). If set to
Returns:
-
Table
–Table from FlatGeobuf file.
read_flatgeobuf_async
async
¶
read_flatgeobuf_async(
path: str,
*,
fs: Optional[ObjectStore] = None,
batch_size: int = 65536,
bbox: Tuple[float, float, float, float] | None = None
) -> Table
Read a FlatGeobuf file from a url into an Arrow Table.
Example:
Reading from an HTTP(S) url:
from geoarrow.rust.core import read_flatgeobuf_async
url = "http://flatgeobuf.org/test/data/UScounties.fgb"
table = await read_flatgeobuf_async(url)
Reading from an S3 bucket:
from geoarrow.rust.core import ObjectStore, read_flatgeobuf_async
options = {
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_region": "..."
}
fs = ObjectStore('s3://bucket', options=options)
table = await read_flatgeobuf_async("path/in/bucket.fgb", fs=fs)
Parameters:
-
path
(str
) –the url or relative path to a remote FlatGeobuf file. If an argument is passed for
fs
, this should be a path fragment relative to the root passed to theObjectStore
constructor.
Other Parameters:
-
fs
(Optional[ObjectStore]
) –an ObjectStore instance for this url. This is required for non-HTTP urls.
-
batch_size
(int
) –the number of rows to include in each internal batch of the table.
-
bbox
(Tuple[float, float, float, float] | None
) –A spatial filter for reading rows, of the format (minx, miny, maxx, maxy). If set to
Returns:
-
Table
–Table from FlatGeobuf file.
read_geojson ¶
Read a GeoJSON file from a path on disk into an Arrow Table.
Parameters:
-
file
(Union[str, Path, BinaryIO]
) –the path to the file or a Python file object in binary read mode.
-
batch_size
(int
, default:65536
) –the number of rows to include in each internal batch of the table.
Returns:
-
Table
–Table from GeoJSON file.
read_geojson_lines ¶
Read a newline-delimited GeoJSON file from a path on disk into an Arrow Table.
This expects a GeoJSON Feature on each line of a text file, with a newline character separating each Feature.
Parameters:
-
file
(Union[str, Path, BinaryIO]
) –the path to the file or a Python file object in binary read mode.
-
batch_size
(int
, default:65536
) –the number of rows to include in each internal batch of the table.
Returns:
-
Table
–Table from GeoJSON file.
read_parquet ¶
read_parquet(
path: Union[str, Path, BinaryIO],
*,
fs: Optional[ObjectStore] = None,
batch_size: int = 65536
) -> Table
Read a GeoParquet file from a path on disk into an Arrow Table.
Example:
Reading from a local path:
from geoarrow.rust.core import read_parquet
table = read_parquet("path/to/file.parquet")
Reading from an HTTP(S) url:
from geoarrow.rust.core import read_parquet
url = "https://raw.githubusercontent.com/opengeospatial/geoparquet/v1.0.0/examples/example.parquet"
table = read_parquet(url)
Reading from a remote file on an S3 bucket.
from geoarrow.rust.core import ObjectStore, read_parquet
options = {
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_region": "..."
}
fs = ObjectStore('s3://bucket', options=options)
table = read_parquet("path/in/bucket.parquet", fs=fs)
Parameters:
-
path
(Union[str, Path, BinaryIO]
) –the path to the file
-
fs
(Optional[ObjectStore]
, default:None
) –the ObjectStore to read from. Defaults to None.
-
batch_size
(int
, default:65536
) –the number of rows to include in each internal batch of the table.
Returns:
-
Table
–Table from GeoParquet file.
read_parquet_async
async
¶
read_parquet_async(
path: Union[str, Path, BinaryIO],
*,
fs: Optional[ObjectStore] = None,
batch_size: int = 65536
) -> Table
Read a GeoParquet file from a path on disk into an Arrow Table.
Examples:
Reading from an HTTP(S) url:
from geoarrow.rust.core import read_parquet_async
url = "https://raw.githubusercontent.com/opengeospatial/geoparquet/v1.0.0/examples/example.parquet"
table = await read_parquet_async(url)
Reading from a remote file on an S3 bucket.
from geoarrow.rust.core import ObjectStore, read_parquet_async
options = {
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_region": "..."
}
fs = ObjectStore('s3://bucket', options=options)
table = await read_parquet_async("path/in/bucket.parquet", fs=fs)
Parameters:
-
path
(Union[str, Path, BinaryIO]
) –the path to the file
-
fs
(Optional[ObjectStore]
, default:None
) –the ObjectStore to read from. Defaults to None.
-
batch_size
(int
, default:65536
) –the number of rows to include in each internal batch of the table.
Returns:
-
Table
–Table from GeoParquet file.
read_postgis ¶
read_postgis_async
async
¶
read_pyogrio ¶
read_pyogrio(
path_or_buffer: Path | str | bytes,
/,
layer: int | str | None = None,
encoding: str | None = None,
columns: Sequence[str] | None = None,
read_geometry: bool = True,
skip_features: int = 0,
max_features: int | None = None,
where: str | None = None,
bbox: Tuple[float, float, float, float] | Sequence[float] | None = None,
mask=None,
fids=None,
sql: str | None = None,
sql_dialect: str | None = None,
return_fids=False,
batch_size=65536,
**kwargs,
) -> RecordBatchReader
Read from an OGR data source to an Arrow Table
Parameters:
-
path_or_buffer
(Path | str | bytes
) –A dataset path or URI, or raw buffer.
-
layer
(int | str | None
, default:None
) –If an integer is provided, it corresponds to the index of the layer with the data source. If a string is provided, it must match the name of the layer in the data source. Defaults to first layer in data source.
-
encoding
(str | None
, default:None
) –If present, will be used as the encoding for reading string values from the data source, unless encoding can be inferred directly from the data source.
-
columns
(Sequence[str] | None
, default:None
) –List of column names to import from the data source. Column names must exactly match the names in the data source, and will be returned in the order they occur in the data source. To avoid reading any columns, pass an empty list-like.
-
read_geometry
(bool
, default:True
) –If True, will read geometry into a GeoSeries. If False, a Pandas DataFrame will be returned instead. Default:
True
. -
skip_features
(int
, default:0
) –Number of features to skip from the beginning of the file before returning features. If greater than available number of features, an empty DataFrame will be returned. Using this parameter may incur significant overhead if the driver does not support the capability to randomly seek to a specific feature, because it will need to iterate over all prior features.
-
max_features
(int | None
, default:None
) –Number of features to read from the file. Default:
None
. -
where
(str | None
, default:None
) –Where clause to filter features in layer by attribute values. If the data source natively supports SQL, its specific SQL dialect should be used (eg. SQLite and GeoPackage:
SQLITE
, PostgreSQL). If it doesn't, theOGRSQL WHERE
syntax should be used. Note that it is not possible to overrule the SQL dialect, this is only possible when you use thesql
parameter.Examples:
"ISO_A3 = 'CAN'"
,"POP_EST > 10000000 AND POP_EST < 100000000"
-
bbox
(Tuple[float, float, float, float] | Sequence[float] | None
, default:None
) –If present, will be used to filter records whose geometry intersects this box. This must be in the same CRS as the dataset. If GEOS is present and used by GDAL, only geometries that intersect this bbox will be returned; if GEOS is not available or not used by GDAL, all geometries with bounding boxes that intersect this bbox will be returned. Cannot be combined with
mask
keyword. -
mask
–Shapely geometry, optional (default:
None
) If present, will be used to filter records whose geometry intersects this geometry. This must be in the same CRS as the dataset. If GEOS is present and used by GDAL, only geometries that intersect this geometry will be returned; if GEOS is not available or not used by GDAL, all geometries with bounding boxes that intersect the bounding box of this geometry will be returned. Requires Shapely >= 2.0. Cannot be combined withbbox
keyword. -
fids
–array-like, optional (default:
None
) Array of integer feature id (FID) values to select. Cannot be combined with other keywords to select a subset (skip_features
,max_features
,where
,bbox
,mask
, orsql
). Note that the starting index is driver and file specific (e.g. typically 0 for Shapefile and 1 for GeoPackage, but can still depend on the specific file). The performance of reading a large number of features usings FIDs is also driver specific. -
sql
(str | None
, default:None
) –The SQL statement to execute. Look at the sql_dialect parameter for more information on the syntax to use for the query. When combined with other keywords like
columns
,skip_features
,max_features
,where
,bbox
, ormask
, those are applied after the SQL query. Be aware that this can have an impact on performance, (e.g. filtering with thebbox
ormask
keywords may not use spatial indexes). Cannot be combined with thelayer
orfids
keywords. -
sql_dialect
–str, optional (default:
None
) The SQL dialect the SQL statement is written in. Possible values:- None: if the data source natively supports SQL, its specific SQL dialect
will be used by default (eg. SQLite and Geopackage:
SQLITE
, PostgreSQL). If the data source doesn't natively support SQL, theOGRSQL
dialect is the default. 'OGRSQL'
: can be used on any data source. Performance can suffer when used on data sources with native support for SQL.'SQLITE'
: can be used on any data source. All spatialite functions can be used. Performance can suffer on data sources with native support for SQL, except for Geopackage and SQLite as this is their native SQL dialect.
- None: if the data source natively supports SQL, its specific SQL dialect
will be used by default (eg. SQLite and Geopackage:
Returns:
-
RecordBatchReader
–Table
write_csv ¶
write_csv(table: ArrowStreamExportable, file: str | Path | BinaryIO) -> None
Write a Table to a CSV file on disk.
Parameters:
-
table
(ArrowStreamExportable
) –the Arrow RecordBatch, Table, or RecordBatchReader to write.
-
file
(str | Path | BinaryIO
) –the path to the file or a Python file object in binary write mode.
Returns:
-
None
–None
write_flatgeobuf ¶
write_flatgeobuf(
table: ArrowStreamExportable,
file: str | Path | BinaryIO,
*,
write_index: bool = True
) -> None
Write to a FlatGeobuf file on disk.
Parameters:
-
table
(ArrowStreamExportable
) –the Arrow RecordBatch, Table, or RecordBatchReader to write.
-
file
(str | Path | BinaryIO
) –the path to the file or a Python file object in binary write mode.
Other Parameters:
-
write_index
(bool
) –whether to write a spatial index in the FlatGeobuf file. Defaults to True.
write_geojson ¶
write_geojson(
table: ArrowStreamExportable, file: Union[str, Path, BinaryIO]
) -> None
Write to a GeoJSON file on disk.
Note that the GeoJSON specification mandates coordinates to be in the WGS84 (EPSG:4326) coordinate system, but this function will not automatically reproject into WGS84 for you.
Parameters:
-
table
(ArrowStreamExportable
) –the Arrow RecordBatch, Table, or RecordBatchReader to write.
-
file
(Union[str, Path, BinaryIO]
) –the path to the file or a Python file object in binary write mode.
Returns:
-
None
–None
write_geojson_lines ¶
write_geojson_lines(
table: ArrowStreamExportable, file: Union[str, Path, BinaryIO]
) -> None
Write to a newline-delimited GeoJSON file on disk.
Note that the GeoJSON specification mandates coordinates to be in the WGS84 (EPSG:4326) coordinate system, but this function will not automatically reproject into WGS84 for you.
Parameters:
-
table
(ArrowStreamExportable
) –the Arrow RecordBatch, Table, or RecordBatchReader to write.
-
file
(Union[str, Path, BinaryIO]
) –the path to the file or a Python file object in binary write mode.
Returns:
-
None
–None
write_parquet ¶
write_parquet(
table: ArrowStreamExportable,
file: Union[str, Path, BinaryIO],
*,
encoding: GeoParquetEncoding | GeoParquetEncodingT = GeoParquetEncoding.WKB
) -> None
Write an Arrow RecordBatch, Table, or RecordBatchReader to a GeoParquet file on disk.
If a RecordBatchReader is passed, only one batch at a time will be materialized in memory.
Parameters:
-
table
(ArrowStreamExportable
) –the table to write.
-
file
(Union[str, Path, BinaryIO]
) –the path to the file or a Python file object in binary write mode.
-
encoding
(GeoParquetEncoding | GeoParquetEncodingT
, default:WKB
) –the geometry encoding to use. Defaults to
GeoParquetEncoding.WKB
.