For most of the past year I’ve been helping build the network for a global, multi-country sporting event — the biggest one anyone’s ever staged. Dozens of venues across a continent. Hundreds of devices. Hundreds of thousands of lines of config, all of which had to be right before the first fan walked through a gate.
At some point I did the math on reviewing it by hand. Eight hours a day of reading configs, every day, and I’d finish one full pass sometime after the closing ceremony — and I’d have caught almost nothing, because nobody notices which of nine hundred nearly identical lines is missing.
So I stopped reading configs and started querying them.
That’s the whole idea behind this series: the network had to become a dataset. Collect everything, normalize it, ask it questions. I don’t think this network gets built any other way — not with more people, and not with more hours.
First, get the data at all
The environment was the kind where the answer to every automation question is no. The only path to the devices was a locked-down jump host. No internet. No scheduled jobs. No automation libraries. No SSH keys. Every “just use an API” answer was off the table on day one.
The workaround was to stop fighting the constraint. I generate terminal-emulator scripts — a scripted version of what a person at a keyboard would type — that crawl the whole fleet through the jump host and capture everything. A simple file bridge carries the captures out to the machine where the analysis runs. The restricted zone only ever sees a plain script and a text file; all the intelligence lives outside it.
It’s low-tech on purpose. That one layer turned “you can’t automate here” into a fresh full-fleet snapshot whenever I wanted one. Every tool in the rest of this post runs on top of it.
The switch someone walked away from
The first real test was dozens of venue core switches that were all supposed to match one reference build. Diffing each one against the golden config doesn’t work — every site legitimately differs in about eighty places, so the real problems drown in expected noise.
The tool parses every config into sections, normalizes the site-specific values away, and scores each switch two ways: against the reference, and against what the majority of its peers are running. The output is a ranked workbook, worst switch first.
The first run flagged a switch missing its router-ID loopback, seventeen telemetry subscriptions, and twenty-odd monitoring lines — a build somebody walked away from in the middle and never came back to finish. The switch was up, passing traffic, green on every dashboard. No amount of reading finds that. The tool found it in seconds and put it at the top of the list with the exact missing lines attached.
No golden config? Let the fleet judge itself
The next question was harder: what’s missing or extra on each switch compared to all the others — with no reference config at all.
The answer was consensus. Union every config element across the fleet; anything on seventy percent or more of the devices is the de facto standard. Then, per device: which standard things are missing, which extra things are present, and where shared config drifts line by line.
That run turned up a whole subset of switches still on an older build template, missing the MPLS and management-hardening pass everyone else had received. Not one finding on one box — a class of inconsistency, identified across the fleet at once, with the member list attached.
The same pass cross-checks every configured IP address against the IP-management database, which produced my favorite finding of the whole build: gateways that disagreed with the live config at every single site. When one device disagrees with the database, the device is probably wrong. When every device disagrees the same way, the database is wrong. You can only see that from the fleet-wide view.
The IP database itself got the same treatment — generated and validated programmatically, then backfilled with hostnames and serial numbers from the monitoring system’s own exports, with a perfect match rate. It stopped being a spreadsheet someone typed and became a mirror of the actual network. That mattered because every other tool checks against it.
Routing config got a structural comparison too, built to separate real divergence from expected per-site differences. That’s the run that caught one of two redundant route reflectors filtering an entire address family toward its clients while its twin didn’t — an asymmetry sitting quietly on the backup control-plane path, waiting for a failover to matter. Nobody reads their way to that one either.
Then keep going
Once the collection layer exists, each new audit is cheap. Per-interface config comparison across the fleet. Discovered neighbor relationships validated against the cabling plan — miscabling found without a truck roll. A management-ACL sweep that found out-of-spec devices and then safely remediated them live. A hunt for ports stuck in a suspended state — the kind of thing that bites on game day. A WAN reachability matrix that turned “is it up?” into a grid. Health reports for the out-of-band paths, so you find out the backup path is dead before the day you need it. Staging new devices became a generated script instead of tribal knowledge.
None of those was a big project. The expensive part — getting trustworthy data out of a locked-down environment into a normalized form — was already paid for.
From reports to dashboards
A report that runs once answers yesterday’s question, and this build changed daily. So the one-shot reports became a pipeline: scheduled collection through the jump host, results into cloud storage, and a dashboard that ingests each run, diffs it against the previous one, and emails when something regresses. A nightly watcher re-runs the validation suite and reports anything new. Config drift, routing comparison, WAN reachability — all of it became something the team glances at, instead of something someone has to remember to run.
That’s also what turned cleanup into a number. The audits had found hundreds of drift items, and “are we getting better?” was a feeling until every finding got scored across crawl rounds: resolved, new, or unchanged. One wrinkle made the scoring interesting — a config class was being deliberately retired, and naive scoring counts planned removals as new drift, which is backwards. It had to learn that decommissioning is progress. With that fixed: about two-thirds of the findings resolved within the first weeks, everything left ranked, and cleanup status became a chart instead of a meeting.
The last box to check before the event was proving the pipes themselves. Dozens of venue servers needed demonstrated bandwidth before real traffic arrived — through the same locked-down access path as everything else. The same scripting harness drove a throughput-test mesh across them and parsed the results into a per-venue baseline: a signed “the network does what we promised” table before day one. It even caught a mistyped test-target address at one venue before it could pollute the numbers.
What made it work
Everything above is the same three moves. Collect everything programmatically. Normalize away the differences that are supposed to exist. Use consensus — against a reference, against the fleet, against a source of truth — to rank what’s left.
That combination beats headcount. More people reading configs mostly produces confident sign-off on the same misses. The half-built switch, the older-template subset, the asymmetric route reflector — none of them get caught by reading harder. They get caught by turning the network into data and asking questions nobody would think to ask, across a fleet nobody can hold in their head.
Then the event started, and the same collection layer built for auditing turned out to be the fastest troubleshooting tool on the network. Traffic that vanished into the core. A dashboard showing a collapse that never happened. A transfer failure that ate weeks until the data cornered it in an hour. Those stories are Part 2.
If you’ve got a system too big to eyeball — a review process that’s really a formality, a question you keep answering by hand — this is the kind of thing I build. Have a look at what I do, or get in touch and tell me what you’ve stopped being able to read.