Part 1 covered the build: the network for a global, multi-country sporting event — dozens of venues, hundreds of devices — was too big to review by hand, so it became a dataset instead. Collect everything through a locked-down jump host, normalize it, ask it questions. The audits that came out of that caught half-built switches, whole subsets on stale templates, and a route-reflector asymmetry nobody would have read their way to.
Then the event started, and the tooling changed jobs.
During the build, a finding meant a task on a punch list. During the event, a finding means something is wrong right now, with a stadium full of people on the network. I half expected the audit tooling to go quiet in this phase. Instead it turned out to be the fastest troubleshooting tool we had, because the questions that matter in an incident — what changed, what’s different about this one device, which layer is lying — are questions the dataset could already answer.
Here’s what that looked like.
One service, one venue
One service VRF at one venue lost internet access. Everything else stayed green: every other VRF at that venue worked, and the same VRF worked everywhere else. A one-cell failure in a grid of hundreds.
A monitoring dashboard caught it inside fifteen minutes — not a user complaint, the dashboard.
The debugging that followed never guessed once. A staged capture script pulled state from each layer in turn, and the comparison data from the build phase ruled layers out one at a time. The venue switch matched fleet consensus, so it wasn’t local config. The route reflectors were advertising what they should, so it wasn’t the control plane. The VPN transport checked out, so it wasn’t the core. About an hour in, exactly one place was left that could drop that one service — a central firewall, where the BGP session for that VRF turned out to be down.
The part I keep thinking about happened the same evening. The monitor that caught the symptom had been watching one thing. By that night it was watching roughly seventy — every service VRF at every venue, tested from inside the VPN, with the target list generated straight from the config crawl. Nobody typed seventy targets into a file. The dataset already knew them.
Symptom in fifteen minutes. Layers eliminated in an hour. Fleet-wide coverage by dinner.
Traffic that vanished into the core
The nastiest one was intermittent. Traffic inside the MPLS core would occasionally vanish — flows blackholed for a stretch, then fine again, with healthy-looking devices on both ends.
The problem with intermittent faults is that the evidence leaves before you arrive. By the time someone logs in, the state that mattered is gone, and you’re staring at a healthy device wondering what you missed. The answer wasn’t cleverness, it was preparation: a capture script, staged in advance, that grabs the full forwarding-plane state of the suspect path in seconds the moment an occurrence starts. Typing those commands by hand takes long enough that the fault heals while you work. The script doesn’t have that problem.
It took more than one occurrence to catch it with full state in hand. Once we had that, the cause fell out: after a supervisor failover on a core device — one configured for graceful restart — some of the label-forwarding state had gone stale. Traffic was following labels toward a path that no longer existed. Everything upstream looked fine, because upstream everything was fine.
The payoff isn’t just the root cause. It’s what got left behind: a runbook, so the next occurrence is a procedure instead of a scramble; the capture script, staged, so catching it in the act takes minutes; and a per-second path-health dashboard — stood up during the incident itself — watching those paths live.
The transfer that only sometimes failed
File transfers from one external partner kept dying mid-session. Everything else was green. This is the classic ticket that eats weeks — “sometimes it works” means every successful retry resets the clock, and no single failure leaves enough evidence to work from. It had, in fact, eaten weeks.
Approached as a data problem, it took about an hour to corner. Staged captures plus the config and routing data already on hand eliminated each layer in turn, same as the VRF hunt. What was left: the flow’s forward path and return path had diverged. An internal routing session had been removed at some point, and after that the traffic went out one way and came back another — with a stateful firewall sitting on only one of the two paths. It saw half of every conversation, and it did what stateful firewalls do with half a conversation: eventually killed it. Mid-transfer.
That’s a named root cause and an identified fix — restore the removed session so the paths converge again — argued entirely from data. Getting a fix through the change process on a network this locked down is its own timeline, and I won’t pretend the ticket closed that afternoon. But there’s a difference between “transfers sometimes die” and “this session was removed, and this firewall now sees half the flow.” One circulates. The other is a change request someone can approve.
Path symmetry had been an assumption. It turned out to be a property you can test for, from data already collected.
The collapse that never happened
One morning a dashboard showed traffic at a core device falling off a cliff. That’s the graph that starts a bridge call.
This time the data’s job was to interrogate the monitoring itself. Before anyone escalated, I pulled the raw poller samples behind the graph and reconstructed what the sampler had actually seen, poll by poll. The dramatic collapse came down to a single poll — one bad sample, stretched into a cliff by the way the graph renders between points. The device was fine. The traffic was fine. The graph was wrong.
Later the same interrogation cut the other way. A quieter change on the same views held up under the same scrutiny — sustained across polls, visible from more than one angle, not an artifact — and that one earned the attention. It’s still being run down, but it’s being run down because the samples say it’s real, not because a graph scared someone. Same method, opposite verdicts: a fake collapse dismissed with evidence, a real change escalated with evidence.
The rest of the shift
Not everything was a mystery. Some of it was just coverage.
Some venues have their own direct internet breakout, which puts them outside the central scrubbing path — an attack on one of those edges wouldn’t show up anywhere anyone was looking until users felt it. The fix was a read-only poller: sample the edge counters over SSH on a schedule and alarm on attack-shaped deltas rather than absolute numbers. Zero config changes on the devices, no change window requested. Visibility at the venues nobody was watching, built without touching anything.
Changes kept landing during the event, and “the change looks fine” is not verification across dozens of devices at two in the morning. So verification became a table: crawl after the change, diff against the state from before it, score every device against the change’s stated intent — took, didn’t take, or side-effect. Known benign mismatches are documented instead of hand-waved. The process even caught a false-positive class in the checker itself, which is exactly the kind of thing you want a checker to admit about its own output.
Underneath all of it, the nightly sentinel from Part 1 kept running: re-run the validation suite, report anything new. During the build, new drift was a to-do item. During the event, new drift is a risk — something changed that nobody announced — and the sentinel notices before people do.
And when a weekend outage did happen, the postmortem came from the same place everything else did. The analysis — including a full port-level map of the inter-core links — was reconstructed from collected state, not from memory or a stale diagram, and the report shipped the same week.
The same data, a different job
Every story above starts at the same place: a current, normalized snapshot of the network that existed before the incident did. Nothing here required building a collection system under pressure. The layer built to audit the network — the one that found the half-built switch and the asymmetric route reflector in Part 1 — is the same layer that found the dead BGP session, the stale labels, the diverged paths, and the poll that lied. That’s the whole series in one line: the dataset that built the network is the dataset that defends it.
If you’re running a system where every incident starts from zero — pull logs, ask around, hope somebody remembers how it’s cabled — this is the kind of thing I build. Have a look at what I do, or get in touch and tell me about the last ticket that ate weeks.