Spatial Dashboard Streamlit · Panel · Production
Caching & Performance

Serializing GeoDataFrames for Redis with WKB

Store the frame as Zstd-compressed Parquet with WKB geometry and the EPSG code in the schema metadata — pickle is two to four times larger, several times slower to load, and breaks on the next GeoPandas upgrade.

Why this matters

A shared cache stores bytes, so every entry crosses a serialisation boundary twice: once on write and once on every hit. For a spatial dashboard that boundary is unusually expensive, because geometry is the bulk of the payload and the obvious approach — pickle.dumps(gdf) — is the worst of the available options on all three axes that matter.

It is large, because pickle stores each Shapely geometry as a serialised Python object with its own header rather than as packed coordinates. It is slow, because reconstructing those objects on read means allocating one Python object per feature. And it is brittle in a way that only shows up later: a pickle embeds the module path and class layout of what it stored, so the day GeoPandas or Shapely is upgraded, every entry written by the old version becomes a deserialisation error on a random worker, at a random time, long after the deploy that caused it.

Well-Known Binary solves all three. It is the standard binary encoding for geometry, it is compact, and it is a format rather than a memory layout, so it is stable across versions. The one thing it does not carry is the coordinate system, which is the subject of the most important step below.

The same layer, three encodingsOne hundred and twenty thousand polygons are serialised three ways and measured on both size and load time. Pickle produces four hundred and twelve megabytes and takes three point one seconds to load, because each geometry becomes a Python object with its own header and each one has to be reconstructed on read. Parquet with the geometry column converted to well-known binary produces one hundred and forty-eight megabytes and loads in eight tenths of a second. Adding Zstd compression brings it to ninety-six megabytes and, counter-intuitively, slightly faster still at six tenths of a second, because the decompression costs less than the extra bytes cost to transfer over the socket. The size axis matters for what fits in the cache instance; the time axis matters more, because it is paid on every hit rather than once per write.120,000 POLYGONS — SIZE AND LOAD TIMEpickle412 MB · 3.1 sParquet + WKB148 MB · 0.8 sParquet + WKB + Zstd96 MB · 0.6 sCompression makes it smaller and slightly faster: the decompression costs less than the extra bytes cost to move acrossthe socket. That trade reverses only for payloads that are already incompressible.The time axis matters more than the size axis — it is paid on every hit, while the size is paid once per write.

Prerequisites

  • geopandas>=1.0, pyarrow>=14, and redis>=5.0.
  • A working shared cache, as set up in Redis as a shared cache layer. This page is the serialisation half of that page in detail.
  • A frame whose CRS is set. A frame with crs=None will round-trip successfully and produce something unusable at the other end.

Step-by-step solution

Step 1 — Convert geometry to WKB and record the CRS

GeoSeries.to_wkb() returns a plain bytes column that Arrow stores as a binary column. The CRS is not part of WKB, so it has to travel separately — the schema metadata is the right place, because it stays attached to the file rather than to a convention both sides have to remember.

python
import io
import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq

def dumps(gdf: gpd.GeoDataFrame) -> bytes:
    if gdf.crs is None:
        raise ValueError("refusing to serialise a frame with no CRS")
    frame = gdf.copy()
    frame["geometry"] = frame["geometry"].to_wkb()
    table = pa.Table.from_pandas(frame, preserve_index=False)
    meta = {**(table.schema.metadata or {}), b"crs": gdf.crs.to_string().encode()}
    table = table.replace_schema_metadata(meta)
    buf = io.BytesIO()
    pq.write_table(table, buf, compression="zstd", compression_level=3)
    return buf.getvalue()

Raising on a missing CRS rather than defaulting to EPSG:4326 is deliberate. A frame that reaches the cache without a coordinate system has a bug upstream, and defaulting hides it until somebody notices that a predicate returns nothing — at which point the cache is several layers away from the actual cause.

Step 2 — Read it back and restore the geometry column

python
def loads(raw: bytes) -> gpd.GeoDataFrame:
    table = pq.read_table(io.BytesIO(raw))
    crs = (table.schema.metadata or {}).get(b"crs")
    if crs is None:
        raise ValueError("cache entry carries no CRS — written by an older writer")
    frame = table.to_pandas()
    frame["geometry"] = gpd.GeoSeries.from_wkb(frame["geometry"])
    return gpd.GeoDataFrame(frame, geometry="geometry", crs=crs.decode())

Both halves raise on the same missing metadata, which means an entry written before this convention existed fails loudly on read rather than producing a frame that silently loses its projection.

Step 3 — Choose the compression level deliberately

Zstd level 3 is the sensible default: it is close to the compression of much higher levels on geometry data and several times faster to write. Levels above about 9 spend meaningful CPU for single-digit percentage gains, which is a poor trade on a path that runs on every cache write.

python
import time

for level in (1, 3, 9, 19):
    t0 = time.perf_counter()
    raw = dumps_at(gdf, level)
    print(f"level {level:2d}: {len(raw)/1e6:6.1f} MB  write {(time.perf_counter()-t0)*1000:5.0f} ms")

Run this once against a representative layer rather than trusting the default. Dense polygon geometry compresses well; a layer that is mostly already-simplified points sometimes barely compresses at all, and there the CPU is being spent for nothing.

Step 4 — Prune columns before serialising, not after

The cheapest bytes are the ones never written. A frame carrying thirty attribute columns of which the dashboard renders three should be pruned on the way into the cache, so every hit for the rest of the entry’s life reads a third of the payload.

python
RENDER_COLUMNS = ["feature_id", "name", "category", "geometry"]

raw = dumps(gdf[RENDER_COLUMNS])

Pruning at write time rather than at read time also makes the cache entry honest about what it holds: an entry that contains only what the map draws cannot accidentally become the source for an export that needed the other twenty-seven columns.

The metadata that has to travel with the bytesA frame is written to the cache and read back. On the upper path the geometry column is converted to well-known binary and the EPSG string is written into the Parquet schema metadata beside it; on read, the geometry is reconstructed from WKB and the coordinate system is restored from the metadata, so the frame that comes out is equal to the frame that went in and every predicate behaves. On the lower path the metadata is omitted — which is what happens with a naive to_wkb and from_wkb pair. The bytes round-trip perfectly, the row count is right, the coordinates are right, and the frame arrives with crs set to None. Nothing raises. The failure appears later and elsewhere, as a spatial join that returns zero rows or an area computed in square degrees, and the cache is several layers away from where anybody will look.WITH THE CRS IN THE SCHEMA METADATAGeoDataFramecrs = EPSG:27700WKB column + b"crs" metadataboth inside the same bytesGeoDataFramecrs = EPSG:27700equalWITHOUT IT — A NAIVE to_wkb / from_wkb PAIRGeoDataFramecrs = EPSG:27700WKB column onlycoordinates, with no systemGeoDataFramecrs = Noneno errorThe row count is right, the coordinates are right, and nothing raises. The failure surfaces later as a join that returnszero rows or an area in square degrees — several layers away from the cache that caused it.Which is why both halves raise on missing metadata rather than defaulting to a plausible EPSG code. What the serialiser must refuseThree inputs should fail loudly at the serialiser rather than quietly at the far end. A frame with no coordinate reference system round-trips perfectly and arrives unusable, so raising is the only way the bug stays near its cause. A frame whose geometry column contains invalid polygons serialises fine and produces predicate errors on read, so validity belongs on the write path where the repair can be logged. A frame carrying columns nobody renders costs its size on every hit for the life of the entry, so an unpruned write is worth rejecting in development even though it is technically correct. Each check is one line, and each converts a delayed, displaced symptom into an immediate one.THREE THINGS TO REFUSE AT THE WRITE PATHa frame with crs = Noneround-trips perfectly and arrives unusable — raise, rather than defaulting to a plausible EPSG codeinvalid geometryserialises without complaint and raises on the predicate several layers laterunpruned columnstechnically correct, and charged on every hit for the life of the entryEach is one line, and each moves a symptom from somewhere distant back to the line that caused it.

Verification

Assert equality on the three things that can silently change: the geometry, the coordinate system, and the dtypes.

python
raw = dumps(gdf[RENDER_COLUMNS])
back = loads(raw)

assert len(back) == len(gdf), "row count changed"
assert back.crs == gdf.crs, "CRS did not survive the round trip"
assert back.geometry.geom_equals(gdf.geometry).all(), "geometry changed"
assert back.dtypes.to_dict() == gdf[RENDER_COLUMNS].dtypes.to_dict(), "dtypes drifted"
print(f"{len(raw)/1e6:.1f} MB for {len(back):,} features")

geom_equals rather than == matters: it compares geometries spatially rather than by object identity, which is the question you actually want answered. And the dtype assertion catches the most common silent regression — a categorical column that comes back as an object column, doubling its memory the moment it is loaded.

Edge cases and gotchas

  • Null geometry. to_wkb() produces None for empty or missing geometries and from_wkb() restores them, so nulls survive — but a frame containing them will fail some predicates downstream. Decide whether to drop them before caching or to keep them, and make it explicit rather than incidental.
  • A mixed-geometry column. WKB handles mixed types fine, but a consumer expecting polygons will not. If the frame is supposed to be homogeneous, assert it before writing rather than discovering it in a renderer.
  • Very large single entries. Redis moves a value as one unit, so a several-hundred-megabyte entry blocks the server while it transfers and delays every other client. Cache per-zoom simplified tiers separately rather than one full-fidelity entry.
  • A schema change without a version bump. Adding a column to RENDER_COLUMNS changes what loads returns, and old entries still hold the old shape. Bump the cache schema version in the key namespace so the shapes cannot mix.

FAQ

Why not use gdf.to_parquet() directly instead of converting to WKB?

You can, and for file storage you should — GeoParquet handles the geometry encoding and the CRS for you. The reason for the manual path here is control over the bytes: to_parquet writes to a path or a file object with its own metadata conventions, and for a cache you want an explicit, versioned encoding whose exact shape you decided, so that a library upgrade changing the default convention cannot silently make existing entries unreadable.

Is WKB lossy?

No. It stores full double-precision coordinates, so a round trip is exact. What it does not carry is the coordinate reference system, the geometry’s validity state, or any Shapely-level attributes — which is why the CRS has to travel in the metadata and why validity should be checked before writing rather than assumed afterwards.

Should I compress before or after Redis?

Before. Redis does not compress values, so an uncompressed payload occupies its full size in the instance’s memory and costs its full size on every transfer. Compressing in the serialiser means the instance holds less and the network moves less, and the decompression cost on read is smaller than the transfer cost it removes.

Back to Redis as a Shared Cache Layer.