Spatial Dashboard Streamlit · Panel · Production
Deployment & Ops

Running Spatial Dashboards on Spot Nodes

A reclaim is a node drain compressed into the warning window, so spot is survivable exactly when the drain hooks already work. Taint the pool, prefer it with weighted affinity, spread across capacity types, and keep enough on-demand to absorb a correlated reclaim.

Why this matters

Spot and preemptible instances are sixty to ninety percent cheaper than on-demand, and a spatial dashboard is an expensive thing to run — the memory required for resident GeoDataFrames pushes it onto larger instance types than a typical web service. The saving is real and the temptation to take it wholesale is strong.

What makes it risky here is not the interruption itself but the session model. A stateless handler that is interrupted mid-request is retried by the load balancer and nobody notices. A dashboard holding WebSocket sessions has no such mechanism: the socket closes, the map goes blank, and the analyst reconnects to a replica that has none of their state. The interruption is not a request failure; it is a visible one, for everybody on that node, at the same moment.

What a spot interruption actually does to a map sessionA spot or preemptible node is reclaimed with a short warning — commonly thirty seconds on one cloud and two minutes on another — and the sequence that follows is the same one a node drain produces, compressed. The node is cordoned so nothing new schedules there, the pods on it are sent a termination signal, and the machine goes away whether or not they finished. For a stateless request handler that is a retry. For a dashboard holding WebSocket sessions it is every open map on that node going blank at once, and the analysts reconnecting to a pod that has none of their state. The warning window is the entire budget for doing something about it, which is why the drain hooks that make a rolling update graceful are the same ones that make spot capacity survivable, and why a grace period longer than the warning is worthless.THE SEQUENCE, AND THE BUDGETreclaim notice — 30 s to 2 minutesthe whole budget; a terminationGracePeriodSeconds longer than this is never honourednode cordonednothing new schedules here, which is the moment the replacement pod should already be startingpods sent SIGTERMthe preStop window runs inside the notice, not after itmachine goneevery session that had not drained is cut, and reconnects to a pod holding none of its stateThe drain hooks that make a rolling update graceful are exactly the ones that make spot capacity survivable — whichis the argument for having them before the cost conversation happens.

Prerequisites

  • The graceful-termination setup from Kubernetes autoscaling & orchestration: a preStop hook, a sensible terminationGracePeriodSeconds, and a PodDisruptionBudget. These are prerequisites rather than refinements — without them spot is simply worse.
  • A reconnect path that rehydrates from session state or from a shared store, so a reconnected analyst lands where they were.
  • Node pools labelled by capacity type, which every managed Kubernetes offering does automatically.

Step-by-step solution

Step 1 — Make on-demand the default

How the scheduler is told which pods may be interruptedThree mechanisms together keep the interruptible half of the fleet on spot and the durable half off it. A node taint on the spot pool means nothing schedules there unless it explicitly tolerates the taint, which makes on-demand the default rather than a preference. Node affinity on the deployment expresses the preference for spot, weighted rather than required, so the pods still schedule when spot capacity is unavailable instead of staying pending. And topology spread constraints across both the capacity type and the zone stop the scheduler from satisfying the affinity by putting every replica in one pool, which would recreate the all-spot failure inside a mixed fleet.TAINT, AFFINITY, SPREADcontrolstaint the spot poolweighted affinitytopology spreadWithout the spread constraint the scheduler happily satisfies a preference for spot by putting every replica there,which turns a mixed fleet back into an all-spot one without anybody changing a percentage.
yaml
# On the spot pool, so nothing lands there by accident.
taints:
  - key: capacity
    value: spot
    effect: NoSchedule

Tainting rather than merely labelling is the important choice. A label expresses a fact and a taint expresses a policy: with the taint in place, a workload ends up on spot only because somebody wrote a toleration for it, which means the decision is visible in the manifest of every service that took it.

Step 2 — Prefer spot, do not require it

yaml
      tolerations:
        - key: capacity
          operator: Equal
          value: spot
          effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80
              preference:
                matchExpressions:
                  - key: capacity
                    operator: In
                    values: ["spot"]

preferred rather than required is what keeps a cost optimisation from becoming an outage. Spot capacity is not always available; a required affinity leaves pods Pending until it is, which means the dashboard is down at exactly the moment the market is tight. Preferred lets the scheduler pay for on-demand instead, which is the correct answer to “spot is unavailable”.

Step 3 — Stop the scheduler from concentrating everything

yaml
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: capacity
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: {app: spatial-dashboard}
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: {app: spatial-dashboard}

Without the first constraint, a weighted preference for spot is satisfied perfectly well by putting every replica on spot — which recreates the all-spot failure inside what the manifest describes as a mixed fleet. The second spreads across zones, because spot reclaims frequently correlate within a zone as well as within a pool.

Step 4 — Size the split against a correlated failure

What fraction of a fleet is safe on spotA dashboard fleet is split between on-demand and spot capacity, and the share of sessions that survive a simultaneous reclaim of the whole spot pool is shown. With everything on spot, a capacity event takes every session at once and the dashboard is down. At seventy-five percent spot, a quarter of capacity remains, which is enough to serve reconnections slowly and not enough to absorb them without queueing. At fifty percent the surviving half absorbs the reconnections within the autoscaler's reaction time. At twenty-five percent spot the reclaim is barely noticeable, and the saving is correspondingly modest. The shape of the curve is the point: spot is not a linear trade of money against reliability, because the failure is correlated — reclaims hit a whole capacity pool at once rather than one node at a time.SHARE OF SESSIONS SURVIVING A FULL SPOT RECLAIM100% spot0% — the dashboard is down75% spot25% remains — reconnections queue50% spot50% — absorbed within the autoscaler's reaction25% spot75% — barely noticeable,and a modest savingSpot failures are correlated: a reclaim takes a capacity pool, not a node. That is why the curve bends rather thansloping, and why a fifty-fifty split behaves so differently from seventy-five.

The number to design against is not the probability of a reclaim but what happens during one. Every session on the reclaimed capacity reconnects at once, and the surviving capacity has to absorb that burst plus whatever load it was already carrying — which is the same burst-headroom problem covered in tuning HPA for bursty sessions, arriving without warning rather than at nine in the morning.

Step 5 — Handle the reclaim notice

yaml
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 20"]
      terminationGracePeriodSeconds: 25    # inside the shortest reclaim notice

Twenty-five seconds looks miserly beside the ninety a rolling update deserves, and it is deliberate: a grace period longer than the reclaim notice is not honoured, because the machine is removed regardless. Size this one against the shortest notice your provider gives and keep the longer period for ordinary drains by using a separate deployment for the on-demand half if the difference matters.

Verification

bash
# 1. Nothing schedules onto spot without a toleration.
kubectl get pods -o wide --field-selector spec.nodeName!="" \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName' | head

# 2. Replicas are actually spread across capacity types.
kubectl get pods -l app=spatial-dashboard -o json \
  | jq -r '.items[].spec.nodeName' | xargs -I{} kubectl get node {} \
  -o jsonpath='{.metadata.labels.capacity}{"\n"}' | sort | uniq -c

# 3. Drain a spot node and watch the sessions.
kubectl drain <spot-node> --ignore-daemonsets --delete-emptydir-data

The third check is the one that matters and the one teams skip. Draining a node deliberately, during a working afternoon, with somebody watching a dashboard session, is the only way to find out whether the reconnect path actually rehydrates — and it is far better to discover it does not while you are holding the keyboard.

Edge cases and gotchas

  • Cluster autoscaler and spot together. A scale-up that requests spot capacity which is unavailable can leave pods pending while the autoscaler retries. Configure the node group to fall back to on-demand, or run a small always-on-demand pool that can absorb the difference.
  • Sticky sessions plus spot. A cookie pins an analyst to a pod that may vanish. That is fine provided the reconnect issues a new cookie and rehydrates — which is the behaviour described in sticky sessions on Kubernetes, and it needs testing under reclaim rather than only under rollout.
  • The reclaim handler is not a shutdown hook. Node-termination handlers cordon and drain on the notice; they do not extend it. Treat them as the thing that starts the drain promptly, not as extra time.
  • Stateful sidecars. A pod with a local cache warmed over an hour loses it on every reclaim. Move that cache to the shared tier, or keep those pods off spot entirely.

FAQ

Is spot worth it for a small fleet?

Often not. Below about four replicas the arithmetic of a correlated reclaim is unforgiving — losing half of four is losing two, and the survivors cannot absorb the reconnections. The saving also matters less in absolute terms. Spot pays off on fleets large enough that a partial reclaim is a degradation rather than an outage.

Should the cache layer run on spot?

No. A shared cache is the thing that makes a reconnected session cheap, so putting it on interruptible capacity means the reclaim takes both the sessions and the mechanism for recovering from it. Keep the shared tier on on-demand and let the application replicas take the risk.

How do I know a reclaim happened rather than a crash?

The node events record it, and the pod’s termination reason differs from an OOM kill or a liveness restart. Log the node name with every session start, as part of the structured logging schema, and a reclaim becomes visible as a cluster of sessions ending together on one node — which is also how you distinguish it from the memory failure it otherwise resembles.

Back to Kubernetes Autoscaling & Orchestration.