Driftwatch
Two numbers from information theory — Shannon entropy and KL divergence — turn raw telemetry into a signal that says "the shape of your traffic just changed." Everything below is interactive, runs offline, and lives in one HTML file. Next page: the signal co-occurrence map → The headline use case is catching a fleet's ticket subject lines when they start to change (sections 2–4); sections 5–7 are the underlying mechanics on other kinds of telemetry.
1 · The two ideas
Shannon entropy — "how spread out is it?"
Low entropy: traffic concentrated on a few values (one hot endpoint, one noisy IP, one error code). High entropy: spread evenly. It tells you how much variety there is, not whether the variety is normal.
KL divergence — "how different from normal?"
P is what you observe now, Q is the baseline. It is the extra bits wasted when you encode live traffic with a model of "normal". It is asymmetric — D(P‖Q) ≠ D(Q‖P) — and blows up when P puts mass where Q says "never".
Metrics like "requests/sec" miss composition changes. A credential-stuffing run can leave total RPS flat while the endpoint mix is wildly off.
A categorical field (endpoint, status, source ASN, error class, node ID), a baseline histogram, and a sliding window of recent counts.
O(N) per window, N = number of categories. A handful of counters per field — cheap enough to run in a sidecar or a log pipeline stage.
2 · Use case: drift in ticket subject lines
A fleet of thousands of servers files thousands of tickets a day. Nobody reads them all, and a new problem rarely announces itself: it shows up as a subject line that wasn't there last week, or an old subject quietly doubling. The idea: treat each ticket's subject as a category, build a histogram over a window, and compare it with a baseline.
Raw subjects are nearly all unique (they contain hostnames and numbers), so counting them says nothing. Collapsing them to templates is what makes the histogram meaningful. Try your own subject:
→
Rules, applied after lowercasing: UUIDs → <id>, IPv4 → <ip>, hostnames (name-1234[.domain]) → <host>, other FQDNs → <fqdn>, percentages → <pct>, versions → <ver>, sizes/durations/decimals → <num>, paths → <path>, integers → <n>. Production systems often use a log-template miner such as Drain; the downstream math is identical.
3 · Live fleet ticket simulator
A simulated fleet of 40 racks files about 3,000 tickets a day across 18 subject templates, with a daily rhythm. The detectors learned a baseline from 14 healthy days and calibrated thresholds on the next 7. Each tick advances simulated hours; the detectors look at the trailing 24 hours. Inject an incident and compare which detector notices, and how fast. Counts get +0.5 smoothing, so a template never seen in the baseline gets a small non-zero q instead of an infinite divergence.
Why is KL high? Top contributing templates (trailing 24 h)
Rack spread = effective number of racks (2^H over racks) for that template's tickets. A few racks means a localized event, ~40 means fleet-wide, in between suggests a hardware or software cohort.
Newest tickets
4 · Try it on your own tickets
Paste subject lines, one per line, oldest first (if a line contains a tab, only the text after the last tab is used). Everything runs locally in your browser; nothing is uploaded. The first part of the list becomes the baseline, the rest is the recent window.
5 · Playground: P vs Q
Five HTTP status buckets. Drag the sliders; values are normalized automatically. Blue = observed P, grey = baseline Q.
Try "Everything is 200": entropy collapses and KL rises. Then try "Mass on a code Q never sees": D(P‖Q) explodes while D(Q‖P) stays modest — that asymmetry is why you almost always want observed ‖ baseline, with smoothing so a zero in Q doesn't give infinity.
6 · Live traffic drift detector
A simulated request stream (200 ms ticks, ~40 requests each) is bucketed by endpoint. The detector keeps a 12-second sliding window, computes entropy and KL against a fixed baseline, and alerts when KL crosses your threshold. Inject an incident and watch which signal notices.
Window vs. baseline, per endpoint
7 · Load balancer evenness
Six backends. Set the share of traffic each receives. For a uniform target U there is a neat identity: D(P‖U) = log₂N − H(P), so entropy and KL are two views of the same balance score.
"Effective nodes" (perplexity, 2^H) reads as: your cluster is behaving as if it had this many equally-loaded nodes. Six nodes with a hot shard may behave like ~2.
8 · Where this fits
| Use case | Categorical field | Baseline Q | What you see |
|---|---|---|---|
| Ticket subject drift | normalized subject template | previous 14 days | New templates flagged NEW, KL names the suspects, rack spread says localized vs fleet-wide; volume often stays silent |
| DDoS / credential stuffing | endpoint, source IP/ASN | same hour last week | H drops, KL rises, top contributor is the targeted path |
| Port scan / enumeration | destination port, URL path | rolling 24 h | H rises (spread out), KL rises on rare buckets |
| Canary / bad deploy | status class, error code | stable fleet | KL(canary‖stable) > 0 before the error-rate alert fires |
| Log anomaly | log template ID (Drain-style parsing) | healthy period | Novel templates produce large KL terms |
| Config / fleet drift | package version, kernel, flag value | golden image | KL per host group points to the drifting cohort |
| Load balance health | backend / shard ID | uniform or weight vector | D(P‖U) = log₂N − H tracks skew |
Practical notes
- Smooth your counts. Use add-α (here α = 0.5) so empty buckets don't yield ∞.
- Window size is a trade-off. Tiny windows have biased entropy (it under-estimates with few samples) and noisy KL; huge windows react slowly.
- Baselines are seasonal. Keep per-hour-of-week baselines, or the daily cycle will look like drift.
- Entropy alone is blind to which values. Swap which endpoint is hot and H stays put; KL does not. Use H for "concentration", KL for "difference", and the per-bucket terms for "who".
- Cardinality. For high-cardinality fields (IPs, user IDs), hash into a fixed number of buckets or keep the top-K plus an "other" bin.
- Threshold from data. Replay history, record the KL distribution on healthy windows, and set the alert at a high percentile of it.