Cross-Region File-Transfer Failures: A Multi-Causal RCA

Problem

External partner organizations moving large files out of a large multi-site enterprise network reported two symptoms. Active-mode FTP hung outright, while passive mode sometimes limped through. And even when transfers worked, everything ran at roughly half the expected speed.

The evidence was contradictory in a way that stalled the whole effort. Enabling the DDoS-mitigation layer had once improved things dramatically — yet a later test showed that disabling it made transfers ten times faster. Both observations were real. Because they seemed mutually exclusive, every team could point at a different culprit: network, firewall, carrier, application. Weeks passed in that loop.

Solution

An evidence-driven root-cause engagement over three sources: a 23,000-line configuration crawl of the edge and core devices, the vendor TAC case history, and the incident channel logs. The analysis was deliberately run from five specialist angles — routing, stateful-firewall behavior, transport/MTU, the DDoS scrubbing layer, and QoS/microburst behavior — then synthesized, together with FTP protocol semantics, into one model.

The conclusion was multi-causal, and that’s the finding that unlocked it:

Symptom A — active-FTP hangs — was stateful-firewall path asymmetry. Forward traffic egressed through one edge; return traffic came back through the other. A server-initiated FTP data connection arrived at a firewall holding no session state for it — and with no FTP ALG to compensate — so it was silently dropped. The configuration fingerprint was decisive: orphaned iBGP route-maps on both edge routers showed that an inter-edge iBGP session used to exist and had been removed. Removing it is what broke path symmetry.

Symptom B — everything slow — was a per-flow tax in the DDoS scrubbing layer (primary), plus a return-side MTU/PMTUD pinch (secondary).

The paradox dissolved: the earlier “scrubbing helped” observation was a path-consistency effect acting on Symptom A; the “scrubbing off is 10× faster” test was removing the Symptom B tax. Two levers, two problems — never one knob.

Deliverables

• Engineer deep-dive — the full evidence chain: configs, route-maps, session behavior, and the reasoning from symptom to cause
• Executive summary — what broke, who’s affected, what fixes it, in plain language for the leadership task force
• One structural fix, staged safely — re-add the inter-edge iBGP session over an interconnect that was already built and addressed, with explicit pre-change verification steps
• A named list of data gaps — the firewall session table and scrubber internals were unavailable; the writeup states what they would confirm rather than guessing

Key Decisions

Refuse the single-root-cause narrative. The evidence didn’t support one cause, and forcing it would have produced a fix that half-worked. Naming two independent problems is what reconciled every observation on record.

Config forensics over live poking. The analysis ran against captured configuration and logs — safe for a locked-down production network — with explicit caveats on what had to be re-verified live before any change window.

Write for both audiences. Engineers got the evidence chain; leadership got a two-page summary. The same finding, two altitudes — which is what ended the finger-pointing.

Name the unknowns. Listing what couldn’t be inspected, and what each item would prove, turned “we’re not sure” into a concrete capture plan instead of an excuse.

Results

• Weeks of contradictory evidence reconciled under one coherent, testable model
• A single highest-leverage structural fix identified — with the config fingerprint to justify it
• Cross-team finger-pointing replaced by an evidence chain each team could verify independently
• Leadership received an actionable plain-language summary instead of a packet-capture debate

How This Scales

This engagement is the template for the Troubleshooting & RCA service: collect the evidence (configs, captures, case history), analyze from multiple specialist angles, synthesize a model that explains every observation, and deliver both the engineer-grade evidence chain and the executive summary — plus a runbook so the same class of issue never needs re-diagnosis.

Tech Stack

• Evidence: full config crawl (23k lines), TAC case digests, incident channel history
• Analysis: BGP/routing forensics, stateful-firewall session semantics, FTP protocol behavior, MTU/PMTUD path analysis
• Tooling: Python monitoring and benchmark harnesses (ping sweep + timed FTP throughput), plus line-level config forensics across devices
• Output: engineer deep-dive + executive summary, staged fix plan with verification gates