Streaming Large GeoJSON Responses from Cloud Run

Return a large FeatureCollection as a chunked, gzipped stream that yields one EPSG:4326 feature at a time, so you never buffer the whole payload past Cloud Run’s 32MiB non-streaming cap or spike memory serializing it all at once.

Jump to heading# Why this matters

A dashboard that serves vector layers from a sidecar API eventually meets two hard walls at the same moment. First, Cloud Run buffers a non-streaming response entirely before sending it and rejects anything over 32 MiB — a moderately dense boundary layer blows past that easily. Second, building that response by calling gdf.to_json() materializes the entire GeoJSON string, plus Python’s intermediate copies, in memory at once, which can be several times the data’s on-disk size and OOM-kill the container. Both problems have the same solution: stop building one giant object. Stream features one at a time with chunked transfer encoding, and the response is bounded neither by the 32MiB cap nor by a memory spike. This is the large-payload counterpart to the Cloud Run deployment workflow, and it pairs naturally with the async data loading patterns the client uses to consume the stream without blocking.

Jump to heading# Prerequisites

• A Cloud Run service that can run a small ASGI/WSGI sidecar next to (or instead of) the dashboard — the base deploy is covered in Cloud Run & Serverless Deployment.
• Python 3.11+, fastapi>=0.110, uvicorn>=0.29, geopandas>=0.14, shapely>=2.0.
• Source data as a GeoParquet or GeoPackage file the service can read, with a known CRS.

Jump to heading# Step-by-step solution

Jump to heading# Step 1 — Build a GeoJSON feature generator

The core idea is a generator that emits valid GeoJSON incrementally: the FeatureCollection preamble, then each feature separated by commas, then the closing brackets. Only one feature’s JSON is ever in memory. Reproject to EPSG:4326 up front because the GeoJSON spec mandates WGS 84 coordinates:

import json
from typing import Iterator
import geopandas as gpd
from shapely.geometry import mapping

def stream_geojson(gdf: gpd.GeoDataFrame) -> Iterator[str]:
    """Yield a valid FeatureCollection one feature at a time (EPSG:4326)."""
    if gdf.crs is None or gdf.crs.to_epsg() != 4326:
        gdf = gdf.to_crs("EPSG:4326")  # GeoJSON requires WGS 84 lon/lat

    yield '{"type":"FeatureCollection","features":['
    first = True
    for row in gdf.itertuples(index=False):
        geom = row.geometry
        props = {k: getattr(row, k) for k in gdf.columns if k != "geometry"}
        feature = {
            "type": "Feature",
            "geometry": mapping(geom),   # shapely -> GeoJSON geometry dict
            "properties": props,
        }
        prefix = "" if first else ","
        yield prefix + json.dumps(feature, default=str)
        first = False
    yield "]}"

Jump to heading# Step 2 — Return it as a chunked streaming response

Wrap the generator in FastAPI’s StreamingResponse. The server flushes each yielded chunk to the wire immediately, so the container never holds the full body and Cloud Run forwards the chunks as chunked transfer encoding:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import geopandas as gpd

app = FastAPI()

@app.get("/layers/{layer}.geojson")
def get_layer(layer: str):
    gdf = gpd.read_parquet(f"data/{layer}.parquet")
    return StreamingResponse(
        stream_geojson(gdf),
        media_type="application/geo+json",
        headers={"Content-Disposition": f'inline; filename="{layer}.geojson"'},
    )

Because the body is streamed, the 32 MiB single-response cap that applies to buffered responses does not bound this endpoint.

Jump to heading# Step 3 — Gzip the stream

GeoJSON is highly compressible — repeated coordinate structure and property keys often shrink by 70–85%. Compress the stream chunk by chunk so the wire payload stays small without ever buffering the whole thing:

import zlib
from typing import Iterator

def gzip_stream(chunks: Iterator[str]) -> Iterator[bytes]:
    """Incrementally gzip a text stream, flushing after each chunk."""
    gz = zlib.compressobj(6, zlib.DEFLATED, zlib.MAX_WBITS | 16)  # gzip header
    for chunk in chunks:
        out = gz.compress(chunk.encode("utf-8"))
        if out:
            yield out
    tail = gz.flush()
    if tail:
        yield tail

@app.get("/layers/{layer}.geojson.gz")
def get_layer_gz(layer: str):
    gdf = gpd.read_parquet(f"data/{layer}.parquet")
    return StreamingResponse(
        gzip_stream(stream_geojson(gdf)),
        media_type="application/geo+json",
        headers={"Content-Encoding": "gzip"},
    )

Jump to heading# Step 4 — Paginate by bounding box

Even streamed, sending the whole planet when the user is looking at one city wastes bandwidth. Filter to the requested bounding box using the spatial index so a request only ever streams the current viewport. Accept the bbox in EPSG:4326 and use gdf.cx for an index-backed slice:

from fastapi import Query

@app.get("/layers/{layer}/bbox.geojson")
def get_layer_bbox(
    layer: str,
    minx: float = Query(...), miny: float = Query(...),
    maxx: float = Query(...), maxy: float = Query(...),
):
    gdf = gpd.read_parquet(f"data/{layer}.parquet").to_crs("EPSG:4326")
    # .cx uses the spatial index — far cheaper than a full intersection scan
    viewport = gdf.cx[minx:maxx, miny:maxy]
    return StreamingResponse(
        gzip_stream(stream_geojson(viewport)),
        media_type="application/geo+json",
        headers={"Content-Encoding": "gzip"},
    )

A client request such as ?minx=-0.51&miny=51.28&maxx=0.33&maxy=51.69 streams only features intersecting Greater London instead of the national dataset.

Jump to heading# Step 5 — Parse incrementally on the client

Consuming the stream feature by feature lets the map paint progressively. A Python client can read the response in chunks; a browser client would use the Fetch streams API the same way:

import requests
import ijson  # incremental JSON parser

def consume(url: str):
    with requests.get(url, stream=True) as resp:
        resp.raise_for_status()
        # ijson yields each object under features[] as it arrives
        for feature in ijson.items(resp.raw, "features.item"):
            yield feature  # render / accumulate as each feature streams in

count = sum(1 for _ in consume("https://svc-xxxx.a.run.app/layers/uk_lad.geojson"))
print(f"streamed {count} features")

Jump to heading# Verification

Confirm the response is genuinely chunked (not buffered) and that peak memory stays flat regardless of feature count:

import requests

resp = requests.get(
    "https://svc-xxxx.a.run.app/layers/uk_lad.geojson.gz", stream=True
)
# Chunked responses carry no fixed Content-Length; they use transfer-encoding.
assert "content-length" not in {k.lower() for k in resp.headers}, \
    "Response is buffered, not streamed — check StreamingResponse wiring"
assert resp.headers.get("Content-Encoding") == "gzip"

total = 0
for chunk in resp.iter_content(chunk_size=64 * 1024):
    total += len(chunk)   # bytes arrive incrementally, not in one buffer
print(f"received {total/1e6:.1f} MB gzipped, streamed in chunks")

Run this against a dataset whose raw GeoJSON exceeds 32 MB: a buffered endpoint would have failed to deploy or errored, while the streamed endpoint returns the full layer cleanly.

Jump to heading# Edge cases and gotchas

• The 32MiB cap only applies to buffered responses. If you accidentally build the full string first — for example wrapping gdf.to_json() in a plain response — you are back under the cap and the memory spike. Verify no fixed Content-Length header is present, which is the tell-tale sign the body was buffered.
• Full serialization memory is the quieter killer. Even below 32 MB on the wire, gdf.to_json() on a large layer can transiently allocate several times the payload size across Python string copies. The generator keeps peak memory flat at one feature, which matters most on the small memory ceilings typical of serverless instances.
• CRS must be EPSG:4326 in the output. GeoJSON coordinates are defined as WGS 84 longitude/latitude. Streaming a layer still in EPSG:3857 or EPSG:27700 yields numerically valid but geographically wrong output. Always to_crs("EPSG:4326") before serializing, and reject or reproject a bbox query whose coordinates are not in degrees.

Jump to heading# FAQ

Back to Cloud Run & Serverless Deployment

Related

• Cloud Run & Serverless Deployment — the deployment workflow and container contract this endpoint runs inside
• Mitigating Cold Starts for Streamlit Spatial Dashboards — sibling technique for keeping the same service responsive
• Async Data Loading Patterns — non-blocking consumption of the streamed feature response
• Memory Limit Management — keeping peak memory flat when serving large spatial payloads