Orchestrating and Autoscaling Spatial Dashboards on Kubernetes

A Streamlit or Panel spatial dashboard behaves nothing like a stateless REST service on Kubernetes. Each browser tab opens a long-lived WebSocket, the server holds the user’s map viewport, filter selections, and often a multi-hundred-megabyte GeoDataFrame in per-session memory, and a single spatial join can pin a CPU core for seconds. Scale the wrong way and you drop active map sessions on every deploy, OOMKill pods when a raster loads, or pay for idle replicas that CPU-based autoscaling refuses to release. This page walks the full orchestration workflow: a Deployment sized for geospatial memory, probes that tolerate a slow geopandas import, a WebSocket-aware Service and Ingress, a HorizontalPodAutoscaler driven by both CPU and connected sessions, and rollout mechanics that never sever a live viewport.

The techniques here build directly on container-level tuning covered in Deployment, Scaling & Production Operations and assume you already ship a working image. They also lean on the session model described in Session State Patterns — because the whole reason a spatial dashboard needs sticky routing and careful drains is that its state lives in the pod, not in a shared store.

Jump to heading# Architecture overview

Autoscaling is a closed control loop, not a one-shot decision. The metrics pipeline samples CPU and a custom active_sessions gauge, the HorizontalPodAutoscaler compares those samples against targets, it writes a desired replica count onto the Deployment, and the Deployment schedules or terminates pods that sit behind a single Service. The diagram below traces that loop and shows where session affinity and disruption budgets clamp its behaviour so scaling never severs a live map.

Jump to heading# Prerequisites

• A Kubernetes cluster 1.27+ with kubectl configured. Managed GKE, EKS, or AKS all work; the manifests below are vanilla and portable.
• metrics-server installed (for CPU-based scaling) and, for the custom-metric step, Prometheus plus either Prometheus Adapter or KEDA 2.14+.
• An ingress controller — the examples use ingress-nginx, which supports cookie affinity and WebSocket upgrades out of the box.
• A working dashboard image exposing Streamlit on 8501 (health at /_stcore/health) or Panel on 5006 (health at /). Container sizing assumes the geopandas>=0.14, shapely>=2.0, pyproj>=3.6 stack and follows the memory guidance in Deployment, Scaling & Production Operations.
• Familiarity with why per-session state matters, covered in Session State Patterns.

Jump to heading# Core implementation workflow

Jump to heading# Step 1 — Write the Deployment with GeoDataFrame-sized resources

Resource requests and limits are the single most consequential setting for a spatial workload. A GeoDataFrame of 200,000 polygons can occupy 300–600 MB once Arrow decodes into Pandas, and CRS reprojection with to_crs() briefly doubles the coordinate array. Under-request and the scheduler packs too many pods per node, so one large query OOMKills a neighbour; under-limit and the kernel kills your own pod mid-render.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spatial-dashboard
  labels:
    app: spatial-dashboard
spec:
  replicas: 3
  selector:
    matchLabels:
      app: spatial-dashboard
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0        # never drop a serving pod during a rollout
      maxSurge: 1              # add one fresh pod at a time
  template:
    metadata:
      labels:
        app: spatial-dashboard
    spec:
      terminationGracePeriodSeconds: 90   # let WebSocket sessions drain
      containers:
        - name: dashboard
          image: registry.example.com/spatial-dashboard:2.4.1
          ports:
            - containerPort: 8501
          env:
            - name: STREAMLIT_SERVER_ENABLE_WEBSOCKET_COMPRESSION
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"      # steady-state residency
            limits:
              cpu: "2"
              memory: "3Gi"      # peak GeoDataFrame + reprojection headroom

Set the memory limit to roughly 2.5–3× the largest single GeoDataFrame the app holds so a reprojection or spatial join never trips the ceiling. Keep the request near steady-state residency so the scheduler bin-packs realistically rather than reserving peak on every node.

Jump to heading# Step 2 — Tune readiness and liveness probes for slow geopandas imports

The geospatial stack imports slowly. Loading geopandas, which pulls in pyproj, shapely, and the compiled GEOS and PROJ libraries, routinely takes 8–20 seconds on a cold pod before Streamlit answers a single request. A naive livenessProbe with a short initialDelaySeconds will declare the pod dead and restart it in a loop before it ever finishes importing. Use a startupProbe to fence off the slow boot, then let liveness and readiness run at a normal cadence.

          startupProbe:
            httpGet:
              path: /_stcore/health
              port: 8501
            periodSeconds: 5
            failureThreshold: 30      # up to 150s for a cold geopandas import
          readinessProbe:
            httpGet:
              path: /_stcore/health
              port: 8501
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3       # pull from Service on sustained failure
          livenessProbe:
            httpGet:
              path: /_stcore/health
              port: 8501
            periodSeconds: 15
            timeoutSeconds: 5
            failureThreshold: 3

Readiness gates traffic: an unready pod is removed from the Service endpoints but not killed, which is exactly what you want while a large layer loads. Liveness only restarts a genuinely hung process. Because startupProbe runs first and suppresses the other two until it passes, the slow import can never trigger a restart. Panel apps use the same shape with port: 5006 and path: /.

Jump to heading# Step 3 — Expose the app with a Service and WebSocket-aware Ingress

Streamlit and Panel push all interactivity over a WebSocket. If the Ingress strips the Upgrade and Connection headers, the dashboard loads once and then freezes — every widget interaction silently fails. The Service is a plain ClusterIP; the Ingress carries the affinity and upgrade configuration.

apiVersion: v1
kind: Service
metadata:
  name: spatial-dashboard
spec:
  selector:
    app: spatial-dashboard
  ports:
    - port: 80
      targetPort: 8501
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spatial-dashboard
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "spatial_route"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  rules:
    - host: dashboard.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: spatial-dashboard
                port:
                  number: 80

The cookie affinity binds each browser to one pod so per-session state stays reachable, and the long proxy-read-timeout stops nginx from killing an idle-but-open WebSocket after the default 60 seconds. The dedicated walkthrough on sticky sessions for stateful spatial dashboards covers the affinity trade-offs and the ClientIP alternative in depth.

Jump to heading# Step 4 — Autoscale on CPU with a HorizontalPodAutoscaler

Start with CPU as the baseline scale-up trigger — it captures the heavy spatial-query bursts well. The critical addition for spatial dashboards is the behavior block: without a scale-down stabilization window, a momentary lull after a spatial join collapses the replica count and disconnects users the moment the next query arrives.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: spatial-dashboard
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: spatial-dashboard
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0        # react fast to query bursts
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300       # wait 5 min before shrinking
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120                # remove at most 1 pod / 2 min

The asymmetry is deliberate: scale up aggressively so a traffic burst never queues behind saturated CPUs, but scale down slowly so a stateful WebSocket dashboard sheds capacity only when the lull is durable.

Jump to heading# Step 5 — Autoscale on active map sessions with custom metrics

CPU alone under-provisions a spatial dashboard because a connected user who is merely reading a map consumes almost no CPU yet still occupies a pod’s memory and one WebSocket slot. The honest signal is the number of active sessions. Emit it from the app, expose it to Kubernetes through Prometheus Adapter, and add it as a second HPA metric.

# metrics_exporter.py — expose active_sessions for Prometheus to scrape
from prometheus_client import Gauge, start_http_server
from streamlit.runtime import get_instance

ACTIVE_SESSIONS = Gauge(
    "spatial_active_sessions",
    "Number of connected spatial dashboard sessions on this pod",
)

def publish_session_count() -> None:
    runtime = get_instance()
    # session_mgr tracks every live WebSocket-backed session on this pod
    ACTIVE_SESSIONS.set(len(runtime._session_mgr.list_active_sessions()))

if __name__ == "__main__":
    start_http_server(9102)   # Prometheus scrapes :9102/metrics

With the metric flowing, extend the HPA to scale on both signals — Kubernetes takes the maximum desired replica count across all listed metrics:

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: spatial_active_sessions
        target:
          type: AverageValue
          averageValue: "40"      # target ~40 live sessions per pod

Tuning these targets and windows for spiky traffic is its own discipline; the tuning HPA for bursty spatial dashboard sessions walkthrough covers stabilization windows, metric lag, and the thundering-herd problem on cache warm-up.

Jump to heading# Step 6 — Protect active sessions with a PodDisruptionBudget

Autoscaling handles the load axis; a PodDisruptionBudget handles the disruption axis. When cluster autoscaling consolidates nodes or an administrator drains a node for a kernel upgrade, Kubernetes will happily evict every dashboard pod at once unless a budget forbids it. For a stateful WebSocket app that eviction means every open map dies simultaneously.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: spatial-dashboard
spec:
  minAvailable: 2          # never drain below 2 serving pods
  selector:
    matchLabels:
      app: spatial-dashboard

minAvailable: 2 guarantees that voluntary disruptions proceed one pod at a time while at least two remain to serve traffic. Combined with the drain hook in the next step, an evicted pod finishes its live sessions before the node goes down.

Jump to heading# Step 7 — Roll out updates without dropping live map sessions

maxUnavailable: 0 from Step 1 already ensures no serving pod is removed until its replacement is Ready. The remaining gap is graceful termination: when an old pod is finally told to stop, its WebSocket sessions must drain rather than snap. Add a preStop hook that flips the pod out of the Service and waits, backed by a terminationGracePeriodSeconds long enough for users to finish.

          lifecycle:
            preStop:
              exec:
                # fail readiness immediately, then let open sessions wind down
                command: ["/bin/sh", "-c", "sleep 60"]

During the preStop sleep the pod stops receiving new sessions (readiness stays green but the rollout has already surged the replacement in) while existing WebSockets keep working. Only after the grace period does the kubelet send SIGTERM. In practice, pair this with a brief client-side reconnect banner so any user who does get bumped reconnects to a fresh pod and rehydrates their viewport from session state.

Jump to heading# Advanced patterns

Jump to heading# KEDA-driven scaling on connected sessions

Prometheus Adapter works, but KEDA expresses session-based scaling more directly and can scale from the metric without maintaining an adapter APIService. A ScaledObject targets the same Deployment and queries Prometheus for the fleet-wide active-session sum:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: spatial-dashboard
spec:
  scaleTargetRef:
    name: spatial-dashboard
  minReplicaCount: 3
  maxReplicaCount: 24
  cooldownPeriod: 300          # match the HPA scaleDown window
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: spatial_active_sessions
        query: sum(spatial_active_sessions)
        threshold: "40"

KEDA synthesises a HorizontalPodAutoscaler under the hood, so the behavior stabilization semantics from Step 4 still apply through its advanced.horizontalPodAutoscalerConfig field.

Jump to heading# Dedicated node pools with more RAM for spatial workers

Spatial pods are memory-heavy and bursty, which makes them poor neighbours for lightweight API pods. Isolate them on a node pool with a higher memory-to-CPU ratio (for example r-family or n2-highmem machines) using a taint on the pool and a matching toleration plus node affinity on the Deployment:

      tolerations:
        - key: "workload"
          operator: "Equal"
          value: "spatial"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: workload
                    operator: In
                    values: ["spatial"]

The high-memory pool absorbs GeoDataFrame spikes without evicting unrelated services, and cluster autoscaling can grow that pool independently when session-based scaling demands more replicas.

Jump to heading# Topology spread for resilience across zones

To survive a zone outage without losing every session, spread replicas across failure domains with a topologySpreadConstraints block. This keeps the fleet from collapsing onto one node or one availability zone when the scheduler bin-packs.

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: spatial-dashboard

maxSkew: 1 keeps zones balanced within one pod of each other, so a single zone failure removes only a fraction of live sessions rather than the whole dashboard.

Jump to heading# Verification and testing

Confirm the HPA sees both metrics and reacts. First inspect its current view:

kubectl get hpa spatial-dashboard --watch
# NAME                REFERENCE                     TARGETS                       MINPODS  MAXPODS  REPLICAS
# spatial-dashboard   Deployment/spatial-dashboard  32%/65%, 18/40               3        20       3

Both targets should render as current/target; if the session metric shows <unknown>/40, the Prometheus Adapter or KEDA wiring is broken and only CPU is driving scale.

Then generate synthetic session load and watch the replica count climb. A small script that opens and holds many WebSocket connections is the honest test, because it exercises the connected-but-idle case that CPU tests miss:

import asyncio
import websockets

async def hold_session(url: str, seconds: int = 300) -> None:
    async with websockets.connect(url) as ws:
        await asyncio.sleep(seconds)   # stay connected, mimic a reading user

async def main(n: int = 200) -> None:
    url = "wss://dashboard.example.com/_stcore/stream"
    await asyncio.gather(*(hold_session(url) for _ in range(n)))

asyncio.run(main())

With 200 held sessions and a target of 40 per pod, the HPA should converge on 5 replicas. Confirm with kubectl get hpa and kubectl get pods -l app=spatial-dashboard, then verify that during the subsequent scale-down no held session drops by watching the client logs — a correctly configured drain keeps every existing connection alive.

Finally, rehearse a rollout under load: run the session holder, then kubectl set image deployment/spatial-dashboard dashboard=registry.example.com/spatial-dashboard:2.4.2 and confirm kubectl rollout status completes with zero client disconnects.

Jump to heading# Troubleshooting

CrashLoopBackOff with Reason: OOMKilled and Exit Code: 137 : The container hit its memory limit, almost always while a large GeoDataFrame loaded or to_crs() doubled the coordinate array. Run kubectl describe pod <name> and look for Last State: Terminated, Reason: OOMKilled. Raise the memory limit to 2.5–3× the largest single layer, move heavy datasets behind a cache, or downcast columns before rendering. Bumping limit without understanding residency only delays the next kill.

Liveness probe failed: Get "http://…/_stcore/health": dial tcp: connection refused during startup, causing repeated restarts : The livenessProbe is firing before the geopandas/pyproj import completes. Add the startupProbe from Step 2 with a failureThreshold high enough to cover the cold import (150 seconds is a safe ceiling). Liveness will not run until the startup probe passes.

WebSocket disconnects the instant the HPA scales down : A pod holding live sessions was terminated. Confirm the HPA behavior.scaleDown.stabilizationWindowSeconds is set (300 is a good default), that a preStop hook and terminationGracePeriodSeconds are present, and that Ingress cookie affinity is enabled so the user is not being reshuffled onto the removed pod. All three from Steps 4–7 must be in place together.

FailedGetPodsMetric: unable to get metric spatial_active_sessions on the HPA : Kubernetes cannot resolve the custom metric. Verify the Prometheus Adapter APIService is Available with kubectl get apiservice v1beta1.custom.metrics.k8s.io, that the exporter is actually being scraped (curl pod-ip:9102/metrics), and that the adapter’s rules map spatial_active_sessions to the Pods metric type.

error: pods "spatial-dashboard-…" is forbidden: unable to fulfil request during a node drain : The PodDisruptionBudget is doing its job — it is blocking an eviction that would drop below minAvailable. This is expected back-pressure, not a bug. Let the autoscaler add replacement capacity first, or drain nodes one at a time so the budget can be satisfied between evictions.

Jump to heading# Performance considerations

Right-size before you autoscale. Autoscaling multiplies whatever a single pod does; if one pod holds a bloated 800 MB layer, ten pods hold 8 GB. Trim columns, downcast dtypes, and simplify display geometry before scaling out — the per-pod savings compound across every replica.

Sessions per pod is the real capacity unit. For a memory-bound spatial dashboard, benchmark how many concurrent sessions one pod serves before p95 interaction latency degrades, then set the HPA averageValue at roughly 70% of that ceiling. This leaves headroom for the reprojection spikes that CPU-based scaling reacts to only after users already feel them.

Match scale-down timing to session length. If a typical analyst keeps a map open for 15 minutes, a 5-minute scaleDown window still risks reclaiming a pod mid-session under affinity. Measure real session duration and set the stabilization window and terminationGracePeriodSeconds against the tail, not the median.

Keep image pulls off the critical path. A cold pod that must pull a multi-gigabyte GDAL-laden image before the slow import even starts blows past probe budgets during a burst. Pre-pull the image onto the node pool or use a DaemonSet warmer so scale-up latency is dominated by the import, not the pull.

Jump to heading# Frequently asked questions

Back to Deployment, Scaling & Production Operations

Related

• Tuning Kubernetes HPA for Bursty Spatial Dashboard Sessions — metric choice, stabilization windows, and anti-flapping policies for spiky session traffic
• Sticky Sessions for Stateful Spatial Dashboards on Kubernetes — cookie affinity, ClientIP trade-offs, and preserving the WebSocket upgrade
• Monitoring & Observability in Production — the metrics pipeline and health signals that feed autoscaling decisions
• Session State Patterns — why per-session pod state is what makes affinity and graceful drains necessary