Profiling GeoDataFrame Memory Usage in Production Dashboards

Measure a GeoDataFrame’s true footprint by combining gdf.memory_usage(deep=True) with the WKB byte size of its geometry column, then confirm against process RSS from psutil — because deep introspection alone undercounts the shapely geometry that lives off the pandas heap.

Jump to heading# Why this matters

Spatial dashboards die from memory in a way tabular ones rarely do. A GeoDataFrame reports a deceptively small memory_usage() because the pandas heap only stores pointers to shapely 2.0 geometry objects, whose coordinate buffers live in native GEOS-backed storage. So the number your profiler prints and the number the out-of-memory killer acts on diverge — sometimes by an order of magnitude on dense polygon layers. Getting this right is the measurement half of Memory Limit Management, and it sits inside the broader work of Caching Strategies & Async Performance Tuning. Once you can attribute bytes correctly, choosing narrower column dtypes — covered in choosing dtypes to reduce GeoDataFrame memory — becomes a measured decision rather than a guess.

Jump to heading# Prerequisites

• Python 3.9+, geopandas>=0.14, shapely>=2.0 (vectorized to_wkb()).
• psutil>=5.9 for resident-set-size snapshots; tracemalloc ships with the standard library.
• A dashboard framework — the gauge example uses streamlit>=1.30.
• A representative layer. The examples read a ~200k-polygon EPSG:4326 building-footprint GeoPackage.

pip install "geopandas>=0.14" "psutil>=5.9"

Jump to heading# Step-by-step solution

Jump to heading# Step 1 — Measure the frame with deep introspection plus geometry

Start with memory_usage(deep=True) for the attribute columns and index, then add the geometry column’s real size separately. sys.getsizeof on the WKB blob approximates the native coordinate storage that deep=True misses.

import sys
import geopandas as gpd


def profile_gdf(gdf: gpd.GeoDataFrame) -> dict[str, int]:
    """Break a GeoDataFrame's memory into pandas-heap and off-heap geometry."""
    per_col = gdf.memory_usage(deep=True)          # bytes per column + index
    geom_pointers = int(per_col.get("geometry", 0))

    # Real geometry storage: shapely 2.0 objects live in native buffers.
    wkb = gdf.geometry.to_wkb()                    # vectorized -> ndarray of bytes
    geom_bytes = int(sum(sys.getsizeof(b) for b in wkb))

    attr_bytes = int(per_col.drop(labels=["geometry"], errors="ignore").sum())
    return {
        "attributes_and_index": attr_bytes,
        "geometry_pointers_reported": geom_pointers,
        "geometry_actual_estimate": geom_bytes,
        "total_estimate": attr_bytes + geom_bytes,
    }


gdf = gpd.read_file("buildings.gpkg")              # EPSG:4326 footprints
report = profile_gdf(gdf)
for k, v in report.items():
    print(f"{k:32s} {v / 1e6:8.1f} MB")

Compare geometry_pointers_reported against geometry_actual_estimate — on dense polygon layers the actual estimate is routinely 5–20× larger, which is exactly the gap that surprises operators.

Jump to heading# Step 2 — Snapshot process RSS with psutil

Per-frame accounting attributes bytes, but only resident set size reflects what the operating system reserves — including native GEOS/GDAL buffers and interpreter overhead. Bracket the load with RSS reads.

import psutil
import geopandas as gpd

proc = psutil.Process()

rss_before = proc.memory_info().rss
gdf = gpd.read_file("buildings.gpkg")
rss_after = proc.memory_info().rss

delta_mb = (rss_after - rss_before) / 1e6
print(f"RSS grew {delta_mb:.1f} MB loading {len(gdf):,} features")
# The RSS delta typically exceeds profile_gdf()['total_estimate'] because it
# also includes GEOS index structures and allocator fragmentation.

Jump to heading# Step 3 — Trace leaks across reruns with tracemalloc

In a long-lived Streamlit or Panel process, the danger is not one large frame but many small ones that never get released. tracemalloc diffs allocation snapshots so you can name the exact line that grows every rerun.

import tracemalloc

tracemalloc.start()
snapshot_a = tracemalloc.take_snapshot()

# ... trigger one dashboard rerun: reload, filter, reproject ...
gdf = gpd.read_file("buildings.gpkg")
subset = gdf[gdf["height_m"] > 30].to_crs(epsg=3857)

snapshot_b = tracemalloc.take_snapshot()
top = snapshot_b.compare_to(snapshot_a, "lineno")
for stat in top[:5]:
    print(stat)          # lines with the largest net allocation between reruns

Run the compare across several reruns. A line whose net size keeps climbing is a retained reference — usually a GeoDataFrame pinned in st.session_state or an unbounded cache. Bound it with max_entries/ttl per Memory Limit Management.

Jump to heading# Step 4 — Emit a live memory gauge

Surface RSS against the container limit so operators see pressure before an OOM kill. A gauge reading 85% of a 2 GB cgroup limit is an early warning, not a postmortem.

import os
import psutil
import streamlit as st


def container_limit_bytes(default=2 * 2**30) -> int:
    """Read the cgroup v2 memory limit, falling back to a sane default."""
    try:
        with open("/sys/fs/cgroup/memory.max") as fh:
            raw = fh.read().strip()
        return default if raw == "max" else int(raw)
    except (FileNotFoundError, ValueError):
        return default


rss = psutil.Process().memory_info().rss
limit = container_limit_bytes()
pct = rss / limit

st.metric("Dashboard RSS", f"{rss / 1e6:.0f} MB", f"{pct:.0%} of limit")
st.progress(min(pct, 1.0))
if pct > 0.85:
    st.warning("Memory above 85% of the container limit — shed cached layers.")

Jump to heading# Verification

Confirm the geometry undercount is real and that your total estimate tracks RSS movement.

import psutil
import geopandas as gpd

proc = psutil.Process()
rss0 = proc.memory_info().rss

gdf = gpd.read_file("buildings.gpkg")
report = profile_gdf(gdf)
rss1 = proc.memory_info().rss

# 1. deep=True materially undercounts geometry
assert report["geometry_actual_estimate"] > report["geometry_pointers_reported"] * 2, \
    "Expected off-heap geometry to dwarf the reported pointer size"

# 2. RSS growth is at least the attribute + geometry estimate (usually more)
rss_delta = rss1 - rss0
assert rss_delta >= report["total_estimate"] * 0.7, \
    "RSS grew far less than estimated — check the layer actually loaded"

print(f"Estimate {report['total_estimate']/1e6:.0f} MB | RSS delta {rss_delta/1e6:.0f} MB")

A healthy result shows the estimate landing within roughly 70–130% of the RSS delta; the native GEOS index and allocator fragmentation account for the overshoot.

Jump to heading# Edge cases and gotchas

• deep=True undercounts shapely geometry. The reported geometry-column size is essentially the pointer array. Always add the WKB estimate from Step 1, and treat process RSS as ground truth for alerting rather than any per-column number.
• Per-session accumulation in Streamlit. Every session holds its own session_state, so ten concurrent users each pinning a 150 MB layer is 1.5 GB before caching even starts. Profile with several sessions open, not one, and prefer a shared cache with bounded eviction over per-session copies.
• Heap fragmentation hides freed memory. Releasing a large GeoDataFrame may not shrink RSS, because glibc malloc retains arenas. RSS staying flat after a del is not necessarily a leak — confirm with tracemalloc (which counts Python allocations) before concluding a reference is retained.

Jump to heading# FAQ

Back to Memory Limit Management

Related

• Memory Limit Management — parent guide to bounding memory footprint in spatial dashboards
• Choosing dtypes to reduce GeoDataFrame memory — narrow column dtypes to act on what profiling reveals
• Caching Strategies & Async Performance Tuning — parent section covering the full performance optimization stack