Monitoring & Observability for Production Spatial Dashboards

A spatial dashboard that renders a 400,000-row GeoDataFrame behaves nothing like a stateless web API when it degrades. Memory climbs quietly as sessions accumulate cached layers, a single slow PostGIS bounding-box query freezes every connected widget, and a tile backend that starts returning 429 responses produces a half-drawn map rather than an obvious error. None of these failures announce themselves in a standard access log. Without deliberate instrumentation the first signal an operator gets is a support ticket that reads “the map is blank,” long after the container has been thrashing at its memory ceiling.

This page builds a complete observability layer for a Streamlit or Panel spatial application from the three pillars of telemetry — metrics, logs, and traces — down to the health probes an orchestrator polls. It covers what to measure, how to expose Prometheus instruments around the spatial query path, how to emit machine-parseable logs, and how to alert on the two failure modes that dominate spatial workloads: out-of-memory kills and slow geoprocessing. It sits under Deployment, Scaling & Production Operations and pairs closely with Kubernetes Autoscaling & Orchestration, whose autoscaler consumes the very metrics defined here.

Jump to heading# The three pillars applied to a spatial dashboard

Observability rests on three complementary data types. Metrics are cheap numeric aggregates sampled over time — a cache hit ratio, a request-duration histogram, resident memory. They answer “is something wrong and how bad.” Logs are timestamped structured events — one record per spatial query with its bounding box, CRS, and row count. They answer “what exactly happened to this request.” Traces stitch a single interaction into a causal timeline of spans — widget callback, cache lookup, database round-trip, geometry serialization — and answer “where did the time go.”

The diagram below shows how these three streams leave the dashboard process, converge on a collector, and fan out to alerting and visualization.

Jump to heading# Prerequisites

Before instrumenting, confirm the following are in place:

• Python 3.10+ with streamlit>=1.32 or panel>=1.3.
• prometheus-client>=0.20 for metric instruments and the exposition endpoint.
• structlog>=24.1 (or the standard library logging module with a JSON formatter) for structured events — the full pattern lives in logging spatial query performance with structured logs.
• opentelemetry-sdk>=1.24 and opentelemetry-exporter-otlp for the tracing section.
• A running spatial backend — PostGIS, DuckDB with the spatial extension, or a tile server — plus familiarity with the query path described in Query Result Caching, since the cache hit ratio is one of the metrics you will export.
• A metrics collector able to scrape an HTTP endpoint and evaluate alert rules.

Install the instrumentation stack:

pip install prometheus-client structlog opentelemetry-sdk opentelemetry-exporter-otlp

Jump to heading# Core implementation workflow

Jump to heading# Step 1 — Decide which spatial signals matter

Generic web metrics (request rate, error rate, p99 latency) are necessary but not sufficient for a spatial dashboard. The signals that actually predict a bad user experience are specific to geospatial workloads. Instrument these six:

The first three are per-query distributions best captured as histograms so you can compute percentiles. The last three are point-in-time values best captured as gauges.

Jump to heading# Step 2 — Expose Prometheus instruments around the spatial query

Define the instruments once at module scope. prometheus_client objects are process-global registries, so defining them inside a function or a Streamlit rerun raises a duplicate-timeseries error. Label only by bounded dimensions — layer name, CRS, and status — never by bounding box or user identity.

from prometheus_client import Counter, Histogram, Gauge, start_http_server
import geopandas as gpd
import time
import threading

# --- Metric definitions (module scope, defined once per process) ---
QUERY_LATENCY = Histogram(
    "spatial_query_duration_seconds",
    "Wall-clock time of a spatial query",
    labelnames=("layer", "crs"),
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)
QUERY_ROWS = Histogram(
    "spatial_query_rows",
    "Row count returned by a spatial query",
    labelnames=("layer",),
    buckets=(100, 1_000, 10_000, 50_000, 100_000, 500_000),
)
CACHE_HITS = Counter("spatial_cache_hits_total", "Cache hits", ("layer",))
CACHE_MISSES = Counter("spatial_cache_misses_total", "Cache misses", ("layer",))
ACTIVE_SESSIONS = Gauge("spatial_active_sessions", "Currently connected sessions")
RSS_BYTES = Gauge("spatial_process_rss_bytes", "Resident set size in bytes")

_metrics_started = False
_metrics_lock = threading.Lock()

def start_metrics_server(port: int = 9095) -> None:
    """Idempotent — safe to call on every Streamlit rerun."""
    global _metrics_started
    with _metrics_lock:
        if not _metrics_started:
            start_http_server(port)   # serves GET /metrics on :9095
            _metrics_started = True

Now wrap the query path. The instrument records latency, row count, and the cache outcome for every call:

def run_bbox_query(layer: str, bbox: tuple[float, float, float, float],
                   crs: str = "EPSG:4326") -> gpd.GeoDataFrame:
    """Fetch features intersecting bbox, fully instrumented."""
    cache_key = f"{layer}:{crs}:{bbox}"
    with QUERY_LATENCY.labels(layer=layer, crs=crs).time():
        cached = _cache.get(cache_key)
        if cached is not None:
            CACHE_HITS.labels(layer=layer).inc()
            return cached
        CACHE_MISSES.labels(layer=layer).inc()
        minx, miny, maxx, maxy = bbox
        gdf = gpd.read_postgis(
            "SELECT geom, name FROM features "
            "WHERE geom && ST_MakeEnvelope(%(minx)s, %(miny)s, %(maxx)s, %(maxy)s, 4326)",
            _engine, geom_col="geom",
            params={"minx": minx, "miny": miny, "maxx": maxx, "maxy": maxy},
        )
        QUERY_ROWS.labels(layer=layer).observe(len(gdf))
        _cache[cache_key] = gdf
        return gdf

The .time() context manager records the observed duration into the histogram automatically, even if the query raises — the timer stops in a finally block internally.

Jump to heading# Step 3 — Sample the process gauges on a timer

Gauges for RSS and active sessions must be refreshed on a schedule because nothing calls them on the request path. A daemon thread sampling once per interval keeps them current without touching the Streamlit script thread:

import os

def _read_rss_bytes() -> int:
    """Read RSS from /proc without a psutil dependency (Linux containers)."""
    with open(f"/proc/{os.getpid()}/status") as fh:
        for line in fh:
            if line.startswith("VmRSS:"):
                return int(line.split()[1]) * 1024   # kB -> bytes
    return 0

def start_resource_sampler(interval_s: float = 10.0) -> None:
    def _loop():
        while True:
            RSS_BYTES.set(_read_rss_bytes())
            time.sleep(interval_s)
    threading.Thread(target=_loop, daemon=True, name="rss-sampler").start()

Increment and decrement ACTIVE_SESSIONS from Streamlit’s session lifecycle: bump it when a new st.session_state is initialised and register an on_session_end callback (or a pn.state.on_session_destroyed hook in Panel) to decrement it.

Jump to heading# Step 4 — Emit structured logs alongside metrics

Metrics tell you the p95 query is slow; logs tell you which query and which bounding box. Configure a JSON formatter so every event is machine-parseable, then log one record per query with the spatial context that a metric label can never safely hold:

import structlog

structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)
log = structlog.get_logger("spatial.query")

def run_bbox_query_logged(layer, bbox, crs="EPSG:4326"):
    t0 = time.perf_counter()
    gdf = run_bbox_query(layer, bbox, crs)
    log.info(
        "spatial_query",
        layer=layer, bbox=list(bbox), crs=crs,
        row_count=len(gdf),
        duration_ms=round((time.perf_counter() - t0) * 1000, 1),
    )
    return gdf

This is deliberately the same query wrapped for a different consumer. The structured logging deep-dive covers correlation IDs, field selection, and how to keep full geometries out of the log stream.

Jump to heading# Step 5 — Add a health endpoint for the orchestrator

Prometheus scraping tells a human when something is wrong; a health endpoint tells the orchestrator whether to keep routing traffic to this pod. Streamlit has no native route, so run a tiny threaded server that reports liveness on /healthz and readiness (data source reachable) on /readyz:

from http.server import BaseHTTPRequestHandler, HTTPServer

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/healthz":          # liveness: is the process up?
            self._respond(200, b"ok")
        elif self.path == "/readyz":         # readiness: can we serve queries?
            ok = _probe_geodata_source()
            self._respond(200 if ok else 503, b"ready" if ok else b"not-ready")
        else:
            self._respond(404, b"not found")

    def _respond(self, code: int, body: bytes):
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass   # silence per-probe access logging

def start_health_server(port: int = 8081) -> None:
    threading.Thread(
        target=HTTPServer(("", port), HealthHandler).serve_forever,
        daemon=True, name="health-server",
    ).start()

The full readiness-probe logic — pinging the spatial database and opening a small GeoDataFrame — is covered in adding health-check endpoints to Streamlit dashboards.

Jump to heading# Step 6 — Alert on OOM and slow queries

Two alert rules cover the failure modes that account for most spatial-dashboard incidents. The first fires as RSS approaches the container limit, giving you a chance to intervene before the kernel OOM-killer terminates the pod:

groups:
  - name: spatial-dashboard
    rules:
      - alert: SpatialDashboardMemoryPressure
        expr: spatial_process_rss_bytes / (2 * 1024 * 1024 * 1024) > 0.85
        for: 2m
        labels: {severity: warning}
        annotations:
          summary: "RSS above 85% of the 2Gi limit"
      - alert: SpatialQuerySlow
        expr: >
          histogram_quantile(0.95,
            sum(rate(spatial_query_duration_seconds_bucket[5m])) by (le, layer))
          > 2.5
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "p95 spatial query latency above 2.5s for {{ $labels.layer }}"

The histogram_quantile function reconstructs the p95 from the histogram buckets defined in Step 2, which is why the bucket boundaries must bracket your latency budget.

Jump to heading# Advanced patterns

Jump to heading# Tracing a bounding-box query with OpenTelemetry

A histogram tells you a query is slow; a trace tells you which stage is slow — the cache lookup, the database round-trip, or the geometry serialization. Wrap the query stages in spans and let a low sample rate keep the overhead negligible:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="collector:4317")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("spatial.dashboard")

def traced_bbox_query(layer, bbox, crs="EPSG:4326"):
    with tracer.start_as_current_span("bbox_query") as span:
        # Attributes are searchable but, like labels, must stay low-cardinality.
        span.set_attribute("spatial.layer", layer)
        span.set_attribute("spatial.crs", crs)
        with tracer.start_as_current_span("db_fetch"):
            gdf = run_bbox_query(layer, bbox, crs)
        with tracer.start_as_current_span("serialize"):
            payload = gdf.to_json()
        span.set_attribute("spatial.row_count", len(gdf))
        return payload

Configure a ParentBased(TraceIdRatioBased(0.02)) sampler so roughly 2% of requests are traced, and add a span processor that force-samples any span whose duration exceeds the budget — slow queries are exactly the ones you want captured.

Jump to heading# RED and USE: two lenses on the same telemetry

The RED method (Rate, Errors, Duration) frames the dashboard as a service: how many spatial queries per second, what fraction error, and how long they take. Those three map directly to the counters and histogram from Step 2. The USE method (Utilization, Saturation, Errors) frames the container as a resource: CPU utilisation, memory saturation (the RSS gauge against the limit), and OOM-kill errors. Run both. RED catches a degrading query path; USE catches a container running out of headroom before RED symptoms even appear. A memory leak shows up in USE (rising saturation) long before it shows up in RED (queries only slow once swapping begins).

Jump to heading# Log sampling for high-traffic layers

A dashboard serving hundreds of tile requests per second per session will drown its log store if every fetch emits a record. Sample the info-level logs but never the errors — a bound structlog processor that drops a fixed fraction of successful events while letting every warning and exception through preserves the diagnostic signal at a fraction of the volume:

import random

def sample_info(_, method_name, event_dict, rate: float = 0.1):
    """Keep all warn/error events; sample info events at `rate`."""
    if event_dict.get("level") in ("warning", "error", "critical"):
        return event_dict
    if random.random() > rate:
        raise structlog.DropEvent
    return event_dict

Jump to heading# Verification and testing

Confirm the exposition endpoint is live and the instruments increment as expected before relying on them in production. Scrape the endpoint directly and assert the metric names appear:

import urllib.request

start_metrics_server(9095)
run_bbox_query("parcels", (-3.20, 55.94, -3.17, 55.96))   # central Edinburgh

body = urllib.request.urlopen("http://localhost:9095/metrics").read().decode()
assert "spatial_query_duration_seconds_bucket" in body, "latency histogram missing"
assert "spatial_cache_misses_total" in body, "cache counter missing"
print("metrics endpoint OK")

Assert the health endpoints answer independently of the Streamlit port:

curl -fsS localhost:8081/healthz && echo " <- liveness ok"
curl -fsS localhost:8081/readyz  && echo " <- readiness ok"

For the cache-ratio metric, drive a cold then warm call and confirm the counters move in opposite directions — a hit on the second call, no new miss:

before_miss = CACHE_MISSES.labels(layer="parcels")._value.get()
run_bbox_query("parcels", (-3.20, 55.94, -3.17, 55.96))   # warm -> hit
after_miss = CACHE_MISSES.labels(layer="parcels")._value.get()
assert after_miss == before_miss, "warm call unexpectedly missed the cache"

Jump to heading# Troubleshooting

ValueError: Duplicated timeseries in CollectorRegistry : A metric was defined more than once — usually because the Counter/Histogram definitions sit inside a function or a Streamlit script body that reruns on every interaction. Move all instrument definitions to module scope so they are created exactly once per process, and guard start_http_server with the idempotent flag shown in Step 2.

OSError: [Errno 98] Address already in use on the metrics port : Streamlit reran the script and called start_http_server a second time. The _metrics_started guard in Step 2 fixes this; if you still see it, another process (or a previous hot-reload) holds the port — confirm with ss -ltnp | grep 9095 and choose a free port or restart the container.

Prometheus time-series count exploding / collector out of memory : A label carries unbounded cardinality. Search your instrument definitions for a bbox, geometry, session_id, or user label. Each distinct value spawns a permanent time series, so a bounding-box label creates effectively infinite cardinality. Remove it from the label set and move that context into the structured logs, which are built for high-cardinality fields.

Metrics endpoint returns an empty body : The instruments exist but were never observed, or you started the server against the wrong registry. Confirm at least one query ran after start_metrics_server, and that you are not passing a custom registry= to the instruments while scraping the default one.

Readiness probe flaps during startup : The /readyz check is running while import geopandas and the GDAL/PROJ shared libraries are still loading, so the spatial-source probe times out and returns 503 even though the process is healthy. Give the probe an initialDelaySeconds longer than the cold-import time, or gate /readyz behind an “imports complete” flag — the pattern is detailed in the health-check endpoints page.

Jump to heading# Performance considerations

Instrumentation is not free, but a well-designed layer costs far less than the incidents it prevents. Keep the overhead invisible with these guardrails:

• Histogram bucket count. Each bucket per label combination is a stored time series. Eight buckets across three layers and two CRS values is 48 series for one histogram — reasonable. Twenty buckets across ten layers is 200 series and climbing. Choose buckets that bracket your latency budget and stop.
• Gauge sampling interval. Reading /proc/self/status every 10 seconds is negligible; reading it on every request adds a syscall to the hot path for no extra insight, since RSS moves slowly.
• Trace sampling rate. Full tracing can add 10–30% CPU on a serialization-heavy path. A 2% parent-based sample with force-sampling for slow queries keeps overhead in the low single digits while still capturing every pathological request.
• Log volume. A JSON record per tile fetch at hundreds of fetches per second per session will dominate your egress bill before it dominates CPU. Sample info logs (as above) and reserve unsampled logging for warnings and errors.

The autoscaler in Kubernetes Autoscaling & Orchestration consumes the RSS gauge and active-session gauge defined here to make scaling decisions, so treat those two metrics as a contract: keep their names and units stable across deploys.

Jump to heading# Frequently asked questions

Back to Deployment, Scaling & Production Operations

Related

• Adding Health-Check Endpoints to Streamlit Dashboards — the threaded liveness and readiness server this page starts in Step 5
• Logging Spatial Query Performance with Structured Logs — the JSON log schema and correlation IDs referenced in Step 4
• Kubernetes Autoscaling & Orchestration — the autoscaler that consumes the RSS and session gauges defined here
• Query Result Caching — the cache layer whose hit ratio is one of the six spatial signals