Choosing Column Dtypes to Reduce GeoDataFrame Memory

Profile with gdf.memory_usage(deep=True), then downcast numerics to 32-bit, turn repeated-string columns into category, and move free text to string[pyarrow] — this routinely cuts a points-of-interest layer by 80% while leaving geometry precision untouched.

Jump to heading# Why this matters

A GeoDataFrame loaded straight from GeoJSON or PostGIS uses pandas’ widest default dtypes: int64 for every integer, float64 for every float, and boxed Python object for every string. On a 400,000-row layer that is hundreds of megabytes of resident memory before a single tile renders — and in a multi-user Streamlit or Panel app that cost is paid per session and per cached copy. Narrowing dtypes is the cheapest, safest optimisation available: it changes storage, not values, so maps and spatial joins produce identical results on a fraction of the RAM. This is the concrete procedure behind the dtype table in the GeoDataFrame Schema Reference, and it is what keeps cached layers under the ceilings in Memory Limit Management.

Jump to heading# Prerequisites

• Python 3.10+ with geopandas>=0.14, pandas>=2.1, and pyarrow>=14.
• A GeoDataFrame you can profile — the example below uses a 400,000-row POI layer in EPSG:4326.

pip install "geopandas>=0.14" "pandas>=2.1" "pyarrow>=14"

Jump to heading# Step-by-step solution

Jump to heading# Step 1 — Profile with memory_usage(deep=True)

Never optimise blind. deep=True walks object and geometry blocks so you see the real footprint, not the shallow pointer estimate.

import geopandas as gpd

gdf = gpd.read_file("pois_global.geojson")   # 400,000 rows, EPSG:4326
print(gdf.dtypes)
# poi_id        int64
# name         object
# category     object
# rating      float64
# geometry   geometry

usage = gdf.memory_usage(deep=True)
print(usage / 1e6)          # megabytes per column
print(f"Total: {usage.sum() / 1e6:.1f} MB")
# Total: 320.4 MB

The two object columns (name, category) dominate — as they almost always do. That tells you where the wins are before you change anything.

Jump to heading# Step 2 — Downcast numeric columns

pd.to_numeric(..., downcast=...) picks the narrowest dtype that holds every value without loss.

import pandas as pd

gdf["poi_id"] = pd.to_numeric(gdf["poi_id"], downcast="integer")  # int64 -> int32
gdf["rating"] = pd.to_numeric(gdf["rating"], downcast="float")    # float64 -> float32

print(gdf["poi_id"].dtype, gdf["rating"].dtype)   # int32 float32

downcast="integer" also chooses int16 or int8 when the range allows, so a resolution or level column can collapse to a single byte per row.

Jump to heading# Step 3 — Convert repeated strings to category

category is the big lever. When distinct values are few relative to row count, each value becomes a small integer code plus one dictionary entry.

n_unique = gdf["category"].nunique()
print(f"category cardinality: {n_unique} / {len(gdf)}")   # 14 / 400000

gdf["category"] = gdf["category"].astype("category")
print(gdf["category"].memory_usage(deep=True) / 1e6)      # ~0.4 MB (was ~96 MB)

A rule of thumb: convert when distinct values are below ~50% of the row count. Fourteen categories across 400,000 rows is a 200x reduction.

Jump to heading# Step 4 — Use the pyarrow string dtype

name is high-cardinality, so category would not help. string[pyarrow] stores the text in one contiguous Arrow buffer instead of one boxed Python str per cell.

gdf["name"] = gdf["name"].astype("string[pyarrow]")
print(gdf["name"].memory_usage(deep=True) / 1e6)          # ~26 MB (was ~88 MB)

Arrow-backed columns also serialize straight into Parquet, so a layer optimised this way is already in the ideal form for a query result cache.

Jump to heading# Step 5 — Keep geometry compact and precise

Do not touch coordinate precision to save memory. Reduce geometry cost only by simplifying display-only layers — which removes vertices, not decimal digits.

# Optional: for display layers only, drop vertices without losing CRS-level accuracy
gdf["geometry"] = gdf.geometry.simplify(0.0001, preserve_topology=True)

final = gdf.memory_usage(deep=True)
print(f"Total: {final.sum() / 1e6:.1f} MB")   # Total: 38.2 MB

Jump to heading# Conversion savings table

The full before/after for the example layer:

Jump to heading# Applying it in one pass

Once you know the target per column, wrap the whole thing in a reusable function so every ingestion path produces the same slim frame. This is the function to call inside a cached loader:

import geopandas as gpd
import pandas as pd

def slim_dtypes(gdf: gpd.GeoDataFrame,
                categoricals: list[str],
                text_cols: list[str],
                cardinality_ratio: float = 0.5) -> gpd.GeoDataFrame:
    """Downcast numerics, categorize low-cardinality labels, Arrow-back text."""
    # Numerics: let pandas pick the narrowest safe width
    for col in gdf.select_dtypes("integer").columns:
        gdf[col] = pd.to_numeric(gdf[col], downcast="integer")
    for col in gdf.select_dtypes("float").columns:
        gdf[col] = pd.to_numeric(gdf[col], downcast="float")
    # Labels: only categorize when values actually repeat
    for col in categoricals:
        if gdf[col].nunique() < len(gdf) * cardinality_ratio:
            gdf[col] = gdf[col].astype("category")
    # Free text: contiguous Arrow buffer instead of boxed objects
    for col in text_cols:
        gdf[col] = gdf[col].astype("string[pyarrow]")
    return gdf   # geometry column deliberately untouched

gdf = slim_dtypes(gdf, categoricals=["category"], text_cols=["name"])

The cardinality_ratio guard is what stops the function from categorizing a name column by mistake — it falls back to leaving high-cardinality columns for the pyarrow pass.

Jump to heading# Nullable integers when data is missing

A plain int32 column cannot hold a missing value; the moment a NaN appears pandas silently promotes it to float64, undoing your downcast. When a count or ID genuinely has gaps, use the nullable Int32 dtype (capital I) so the integer width — and the saving — survives the null:

gdf["visits"] = gdf["visits"].astype("Int32")   # holds <NA> without float coercion

Jump to heading# Verification

Assert both the memory target and that values are unchanged.

# Capture originals BEFORE the casts above to compare against
original_categories = gpd.read_file("pois_global.geojson")["category"].tolist()

# Memory target met
total_mb = gdf.memory_usage(deep=True).sum() / 1e6
assert total_mb < 45, f"Still {total_mb:.1f} MB — check for stray object columns"

# Values preserved (dtype change must not alter data)
assert gdf["category"].astype("string").tolist() == original_categories
assert gdf.crs.to_epsg() == 4326          # CRS untouched
assert set(gdf.geom_type.unique()) == {"Point"}
print(f"Optimised to {total_mb:.1f} MB with values intact")

If a supposedly-optimised frame is still large, memory_usage(deep=True) will point straight at the remaining object column.

Jump to heading# Edge cases and gotchas

• category on high-cardinality columns backfires. If distinct values approach the row count, the dictionary plus the integer-code array is larger than a plain string[pyarrow] column. Always check nunique() first; reserve category for labels with genuine repetition.
• float32 loses coordinate precision — never downcast geometry. float32 carries ~7 significant digits; at global longitude that is roughly metre-level, and reprojection amplifies the drift into visible misalignment. Downcast attribute floats only, and keep the geometry column at native double precision.
• pyarrow interop is attribute-only. Arrow-backed attribute columns coexist with a normal geopandas GeoSeries, so spatial predicates and to_parquet keep working — but some third-party libraries still expect NumPy-backed columns, so cast back with .astype("float64") at the boundary of any library that errors on Arrow input.

Jump to heading# FAQ

Back to GeoDataFrame Schema Reference

Related

• GeoDataFrame Schema Reference — the full dtype table and schema-validation workflow this procedure implements
• Converting shapely geometries to GeoJSON for web maps — exporting a slimmed layer for the browser
• Memory Limit Management — container ceilings that a slimmed layer helps you stay under
• Query Result Caching — why Arrow-backed, downcast frames cache deterministically