Sharing One Cache Across Every Worker With Redis

Every caching decorator this site has covered so far caches inside a process. That is exactly right for a single container serving a handful of analysts, and it stops being right the moment a second replica appears. Two workers behind a load balancer hold two independent caches, so the same expensive PostGIS query runs once per worker; four workers make it four times; an autoscaler that adds a replica during a burst hands the new pod an entirely cold cache at precisely the moment the system is least able to absorb the extra load. A shared cache moves the stored result out of the process and in front of all of them.

Redis is the usual answer because it is the one piece of infrastructure that is already in most stacks, but the interesting part is not the client library — it is that a shared cache changes what a cache entry is. In-process, an entry is a live Python object; shared, it is a sequence of bytes that has to survive serialisation, a network hop, an eviction policy you do not fully control, and being read back by a worker running a possibly different version of your code. This page is about getting those four things right for geospatial payloads specifically, where entries are large, geometry does not serialise for free, and a careless key design multiplies entries faster than any eviction policy can remove them.

Jump to heading# The problem in one number

A dashboard serving eight replicas, each holding a 300 MB in-process cache of the same twelve administrative layers, is spending 2.4 GB of cluster memory to store 300 MB of data eight times. It is also issuing eight cold-start query storms every deploy, and giving eight different answers for the few minutes after an upstream refresh, because each worker’s entries expire on their own schedule. Consolidating those entries into one Redis instance removes the duplication, makes invalidation a single operation rather than eight independent ones, and — the part that is easy to miss — makes cache behaviour reproducible, because there is now exactly one place to look when a stale answer appears.

Jump to heading# Prerequisites

• redis>=5.0 as the Python client, and a Redis 6 or 7 server reachable from every replica. A managed instance is fine; what matters is that it is one instance, or one cluster, and not one per pod.
• geopandas>=0.14 and pyarrow>=14, because the serialisation format below is Parquet with WKB geometry rather than pickle.
• A deterministic cache key, as covered in Query Result Caching. A shared cache makes key design more important, not less: a key that varies per process was merely wasteful before and is now actively confusing, because two workers will disagree about whether an entry exists.
• Enough memory on the Redis instance for the working set, plus headroom. Sizing it is the subject of a later section, and guessing is the most common way this ends badly.

Jump to heading# Core implementation workflow

Jump to heading# Step 1 — Serialise to bytes, not to pickle

A shared cache stores bytes. The temptation is to reach for pickle, because it round-trips any Python object with one call, and it is the wrong choice here for three separate reasons: it is large for geometry, it is slow to load for large frames, and it couples the cache to the exact class layout of the objects that wrote it, so a GeoPandas upgrade can make every existing entry unreadable in a way that surfaces as a deserialisation exception on a random worker.

Parquet with WKB geometry avoids all three. It is a columnar format that compresses geometry well, it loads quickly, and it is a stable on-disk format that any version of the library can read.

import io
import geopandas as gpd
import pyarrow.parquet as pq
import pyarrow as pa

def to_bytes(gdf: gpd.GeoDataFrame) -> bytes:
    """Serialise a frame for the shared cache: WKB geometry, Zstd-compressed Parquet."""
    frame = gdf.copy()
    crs = frame.crs.to_string() if frame.crs else None
    frame["geometry"] = frame["geometry"].to_wkb()
    table = pa.Table.from_pandas(frame, preserve_index=False)
    table = table.replace_schema_metadata({**(table.schema.metadata or {}),
                                           b"crs": (crs or "").encode()})
    buf = io.BytesIO()
    pq.write_table(table, buf, compression="zstd")
    return buf.getvalue()

def from_bytes(raw: bytes) -> gpd.GeoDataFrame:
    table = pq.read_table(io.BytesIO(raw))
    crs = (table.schema.metadata or {}).get(b"crs", b"").decode() or None
    frame = table.to_pandas()
    frame["geometry"] = gpd.GeoSeries.from_wkb(frame["geometry"])
    return gpd.GeoDataFrame(frame, geometry="geometry", crs=crs)

Carrying the CRS in the schema metadata is not optional. Geometry serialised to WKB loses its coordinate system, and a frame that comes back from the cache without one is the exact failure mode described in CRS & coordinate systems: every subsequent predicate silently returns nothing.

Jump to heading# Step 2 — Namespace the key with a schema version

A shared cache outlives any single deploy, which means the code that reads an entry may not be the code that wrote it. Put a version in the key namespace and bump it whenever the serialised shape changes — a renamed column, a different simplification tolerance, a change to which columns are stored.

import hashlib
import redis

CACHE_SCHEMA = "v3"          # bump on any change to what to_bytes() produces
_pool = redis.ConnectionPool.from_url(REDIS_URL, max_connections=32)

def _client() -> redis.Redis:
    return redis.Redis(connection_pool=_pool)

def cache_key(layer: str, bbox: tuple[float, float, float, float], epsg: int) -> str:
    """Namespaced, quantised, and stable across processes."""
    rounded = tuple(round(v, 3) for v in bbox)          # ~110 m grid, kills float drift
    digest = hashlib.sha256(f"{layer}|{rounded}|{epsg}".encode()).hexdigest()[:24]
    return f"spatial:{CACHE_SCHEMA}:{layer}:{digest}"

Bumping the version does not delete the old entries — it orphans them, and their own TTL removes them. That is deliberate: a deploy that had to flush the cache synchronously would take the shared cache down for every replica at once, whereas orphaning lets the old and new shapes coexist for a few minutes while the rollout completes.

Jump to heading# Step 3 — Wrap the read in a get-or-compute helper

from typing import Callable

def cached_frame(key: str, compute: Callable[[], gpd.GeoDataFrame],
                 ttl: int = 900) -> gpd.GeoDataFrame:
    """Return the cached frame, computing and storing it on a miss."""
    r = _client()
    try:
        raw = r.get(key)
        if raw is not None:
            return from_bytes(raw)
    except redis.RedisError:
        pass                                  # a cache outage must not be an outage

    frame = compute()
    try:
        r.set(key, to_bytes(frame), ex=ttl)
    except redis.RedisError:
        pass
    return frame

The two bare except clauses are the most important lines on this page. A shared cache is a piece of infrastructure that can be unreachable, full, or restarting, and none of those should turn into a five-hundred error for an analyst who only wanted a map. Degrading to the uncached path is slower and correct; propagating the exception is neither.

Jump to heading# Step 4 — Guard the stampede

When a popular entry expires, every replica misses it at the same moment and every replica runs the same expensive query. With eight workers that is eight identical PostGIS queries arriving simultaneously, which is the load spike the cache existed to prevent, and it arrives on a schedule set by the TTL.

A per-key lock reduces that to one. The first worker to claim the lock computes; the others wait briefly and then read the value the winner wrote.

import time

def cached_frame_guarded(key: str, compute, ttl: int = 900,
                         lock_ttl: int = 30) -> gpd.GeoDataFrame:
    r = _client()
    raw = r.get(key)
    if raw is not None:
        return from_bytes(raw)

    lock = f"{key}:lock"
    if r.set(lock, b"1", nx=True, ex=lock_ttl):    # we won: compute and publish
        try:
            frame = compute()
            r.set(key, to_bytes(frame), ex=ttl)
            return frame
        finally:
            r.delete(lock)

    for _ in range(int(lock_ttl / 0.25)):          # we lost: wait for the winner
        time.sleep(0.25)
        raw = r.get(key)
        if raw is not None:
            return from_bytes(raw)
    return compute()                               # winner died — compute anyway

The final return compute() matters. A lock holder that is OOM-killed mid-query would otherwise leave every other worker waiting for a value that will never arrive; falling through to an uncached computation is the behaviour that keeps the dashboard responsive during exactly the incident that is hardest to reproduce.

Jump to heading# Step 5 — Keep a small local tier in front of it

A Redis hit is fast, but it is not free: it costs a round trip plus a deserialisation, and for a large frame the deserialisation dominates. Within a single rerun a dashboard may ask for the same layer several times, and paying that cost repeatedly is pure waste. Keep the framework’s in-process cache in front of the shared one, bounded tightly, so the shared cache absorbs cross-worker duplication while the local cache absorbs within-request duplication.

import streamlit as st

@st.cache_data(ttl=300, max_entries=6, show_spinner=False)
def get_layer(layer: str, bbox: tuple, epsg: int) -> gpd.GeoDataFrame:
    """Local tier in front of the shared tier: two levels, one key function."""
    key = cache_key(layer, bbox, epsg)
    return cached_frame_guarded(key, lambda: query_postgis(layer, bbox, epsg))

max_entries=6 is deliberately small. The local tier exists to hold what this session is looking at right now; everything else lives in Redis, where it is shared. A large local tier would recreate the duplication the shared cache was introduced to remove.

Jump to heading# Advanced patterns

Invalidating by pattern, carefully. When an upstream dataset refreshes, the entries derived from it are stale regardless of their TTL. It is tempting to reach for KEYS spatial:v3:parcels:* and delete the matches — and KEYS blocks the Redis server while it scans the entire keyspace, which on a large instance is long enough to time out every other client. Use SCAN with a cursor and a bounded count, or, better, avoid the problem: put the dataset version in the key as the query result caching page describes, so a refresh changes the key rather than requiring a deletion.

Compressing selectively. Zstd is fast enough that compressing everything is a reasonable default, but the ratio varies enormously by payload: dense polygon geometry compresses well, already-simplified point layers barely at all, and a frame that is mostly a categorical column compresses to almost nothing. Where a payload is both large and incompressible, the compression is a CPU cost paid on every write for no benefit — measure before assuming.

Read replicas for read-heavy layers. A dashboard whose cache is 95% reads can serve those reads from a Redis replica while writing to the primary. The subtlety is that replication is asynchronous, so a worker that writes an entry and immediately reads it back from a replica can get a miss. Route reads for stable reference layers to the replica and reads on the write path to the primary, rather than sending everything to the replica and being surprised.

Jump to heading# Verification and testing

Confirm that the shared cache is actually shared, that a hit is genuinely cheaper than the query, and that a Redis outage degrades rather than fails.

import time

# 1. Two "workers" — two clients — see one entry.
key = cache_key("parcels", (-0.51, 51.28, 0.33, 51.69), 4326)
_client().delete(key)
a = cached_frame_guarded(key, lambda: query_postgis("parcels", BBOX, 4326))
assert _client().exists(key) == 1, "the entry was never published"
t0 = time.perf_counter()
b = from_bytes(_client().get(key))
hit_ms = (time.perf_counter() - t0) * 1000
assert len(a) == len(b) and a.crs == b.crs, "round trip changed the frame"
print(f"hit: {hit_ms:.0f} ms   payload: {len(_client().get(key)) / 1e6:.1f} MB")

# 2. A cache outage must not be an outage.
broken = redis.Redis(host="127.0.0.1", port=1, socket_connect_timeout=0.2)
assert len(cached_frame(key, lambda: query_postgis("parcels", BBOX, 4326))) > 0

The CRS assertion in the first block is the one that catches the most defects. A round trip that loses the coordinate system produces a frame that looks correct in head(), has the right row count, and returns nothing from every spatial predicate applied to it afterwards.

Jump to heading# Troubleshooting

redis.exceptions.ConnectionError: Too many connections. Each worker process is opening its own pool and the pools together exceed maxclients. Set max_connections on the pool to a value that, multiplied by the replica count, stays comfortably under the server limit, and create the pool at module scope so it is shared by every thread in the process rather than per request.

Entries vanish long before their TTL. The instance is at its memory limit and the eviction policy is removing them. INFO memory will show evicted_keys climbing. Either the working set is larger than the instance, or maxmemory-policy is set to something that ignores TTLs — allkeys-lru is usually what a cache wants, and noeviction turns a full instance into write errors instead.

pyarrow.lib.ArrowInvalid on read after a deploy. The serialised shape changed without the schema version being bumped, so new code is reading entries written by old code. Bump CACHE_SCHEMA and let the orphaned entries expire.

Every read is a miss, and the keys look identical. They are not identical — they differ in a way the eye skips. The usual causes are an unrounded float in the bounding box, a CRS given as 4326 in one path and "EPSG:4326" in another, or a dictionary whose iteration order differs between processes. Log the key alongside the hit or miss for one session and compare two that should have matched.

Latency is fine locally and terrible in production. The Redis instance is in a different availability zone from the workers. A cross-zone round trip of a few milliseconds is irrelevant for a small value and dominant for a fifty-megabyte frame; co-locate the cache with the workers that read it.

Jump to heading# Performance considerations

Size the instance against the working set rather than the total. The working set is the number of distinct layer-and-viewport combinations actually requested in a TTL window, which is usually far smaller than the number that could exist — and it is measurable: log the distinct keys requested over an hour and multiply by the mean payload size. Add headroom of at least fifty percent, because a shared cache that spends its life at the eviction threshold behaves like no cache at all while costing a network hop.

Watch the payload size distribution rather than the mean. Redis handles many small values well and a few very large ones poorly; a single entry of several hundred megabytes will block the event loop while it is transferred, delaying every other client. Where a layer is that large, the answer is not a bigger instance but a smaller entry — simplify per zoom tier and cache the tiers separately, as caching strategies describes.

Finally, measure the hit path against the compute path honestly. A shared cache is worth having when deserialising the entry is meaningfully cheaper than recomputing it, and for a small query against a well-indexed table it sometimes is not. The layers worth putting in Redis are the ones whose computation is expensive and whose result is shared across sessions — not every frame the dashboard touches.

Back to Caching Strategies & Async Performance Tuning.

Jump to heading# Related

• Query Result Caching
• Memory Limit Management
• @st.cache_data Implementation
• Kubernetes Autoscaling & Orchestration