That MSTP From Hell Story

Yet another story I tell all the time at hackercons, now in full textual glory.

As many of you know, I spent a good chunk of time working for a WISP. That is Wireless Internet Service Provider for you non technical civilians. I used to refer to this org either by name, or as $weMicrowavePackets. Get it? We used microwaves to send packet data? It’s funny, it’s a funny joke, laugh. Anyways, one of the many, many, many reasons I left was my sheer disgust at just how bad our service had gotten in my final years. We were so completely oversubscribed. The customers complained to the customer service group constantly, who in turn came to me, the senior network engineer, for a solution. There was none, we were oversubscribed, simple fact. We were oversubscribed, and I did not have the authority nor the budget to do anything about it. After I left, the name $weMicrowavePackets died, replaced by the far less flattering moniker $weDropPackets. Believe you me, in the last years I was there, we did more than our fair share of it.

The wireless network was essentially divided into two pieces: East and West. They were connected by way of a single 1.7 gbit wireless link, and later, a ring of mostly 10 gbit fiber was added. Sadly, one expensive as hell 1 gbit loop meant the East side of the network only had a theoretical max of 2.7 gbit bandwidth to the outside world. We had two 10 gbit upstream circuits, as well as 10 gbit to a CDN aggregation network that included Netflix, but none  of that is where we had bottlenecks.

This meant roughly 50% of our customers had access to roughly 12.5% of our available bandwidth. In short, the entire eastern half of the network was oversubscribed right at its headend. This had the effect of making it far easier to deal with, despite generating just as many customer complaints as its western sister net. None of the links leading up to the headend, save the penultimate hops, were generally over saturated. All the traffic had one way out, and every path to that egress sucked because the egress itself sucked. So, at least it sucked consistently. This actually made my life significantly easier.

Contrast this with wireless zone west, which had 10 gbit fiber to 20 gbit of available uplink. It should go without saying at this point that price, not utility, determined everything.

Balancing L3 traffic between these two zones and our upstream providers was also a lot of fun, we had a /19 and a /22 of public address space. A lot of that was announced as /23s in various states of ASN prepend, and was changed on a fairly regular basis. That madness is another entire rant unto itself.

Both sides of the network had evolved organically out of a significantly smaller network of the same core design. The earliest stages worked reasonably well, but they did not scale. In the earlier era, east and west were actually one large mesh. As we added new tower sites, the scale issue came to a head. I had advocated a switch from MSTP (multi spanning tree protocol) to MPLS/VPLS very early on, but was consistently shot down.

Side note: Apparently, to this day, MPLS is considered some kind of Juniper/Cisco propaganda at $weDropPackets. They believe it doesn’t actually scale. I have since worked for networks 10 times the size of $weDropPackets that were near 100% MPLS, trust me, it scales just fine. You can thank a disparaging 2001ish NANOG comment by Randy Busch for the owner’s attitude toward MPLS. It isn’t like technologies mature over time.

So the network lived on in two pieces, as a near unmanageable MSTP mess. A real network engineer will tell you that a backup link has to be able to absorb all the traffic from the primary, or it isn’t a very good backup. A owner/manager concerned with bottom line will insist that all potential links should run balls to the wall, because, “stuff rarely breaks and traffic is money!” Unfortunately, near as I can tell, MSTP was created for managers with this exact mindset. If you are so oversubscribed that you need to split up your layer two paths, damn good possibility you have a massive problem.

It wasn’t my design, it wasn’t what I wanted, but it was my problem. I had to keep this thing running. My name was Sisyphus and this  was my boulder. I was, on paper, the senior network administrator. In reality, I was that so long as my boss hadn’t had a bad day. If he was in a foul mood, all bets were off. He became a micromanaging asshat who would make undocumented changes, ignore change controls, and say “Fuck” an awful lot. Then the undocumented mess he had created would piss him off because our documentation sucked, and the cycle would begin again. Keeping him in a less than bad mood actually became a network stability issue toward the end. Not a pretty picture is it? This was my job, seven days a week. The Internet never sleeps.

So, the western wifi zone. Sixteen major tower sites, and a dozen or so minor ones, and one, singular, way, out. Major sites had links to more than just one other tower site. Minor towers were stubs, they had one uplink to a major site. The end result was a “core” network of 20 or so wireless links, between 16 sites, and all traffic going toward, and coming from, a single headend tower.

This was all managed via MSTP, it was all layer 2, strictly Ethernet.

Let that sink in. Link speeds between 700 mbit, and 35 mbit. Sixteen sites. Twenty something major connections between them. All MSTP. No end in sight, and the owner thought this was just fine. My job was to keep it alive. My solution was horrifying, but I stand by it as the only reasonable course of action left to me.

From memory, I have pieced together what I can remember this network looked like, I cannot seem to recall two of the towers, but this is pretty accurate from the 14 I can recall:

I’m missing a few towers and links, but that should illustrate the horror that was the western network. And yes, LU to LE had two links. Many of these links are actually dual wireless in a link aggregation, but this particular one was of two wildly different speeds. That doesn’t aggregate well at all, so they were handled as separate entities.

Also, this is not to any scale, these links existed mostly as geography dictated. Graphviz just built its own layout.

We had already deployed MSTP in the form of two instances, plus the common. When we split the network in half, that became four instances, plus two commons. That worked for a while.

Then, the over-subscription got really bad.

The western wifi zone effectively had, at least at the headend, 10gbit of bandwidth for apx 50% of the customer load. But the links leading into the headend, numerous that they were, were in total, no more than four gigabit in total capacity. To add to the hell, none of the links were of the same capacity, nor did we have any way of dealing with overloaded links in an automated way. Remember when I said I had advocated MPLS? Now I was outright begging for it. I even built the core network around it, at least in name. Switch management networks (layer 3 switches whenever I won that argument) were all OSPF connected, and named things like, “WifiWest-mpls-105. WifiWest was the network label, mpls was the predominant purpose of the network, and 105 was the VLAN tag. I was building the capability to switch to MPLS in multiple stages. Network tests indicated we could deliver (in lieu of capacity issues), line rate L2 services between wireless east and west. MPLS was never deployed, we instead deployed a “carrier grade Ethernet” solution that was awful on many, many levels.

We also had a pile of aging Cisco L2 switches, some of which did not speak industry standard MSTP. They spoke Cisco’s pre standard to the spec only. We had become, under my direction, a Juniper shop. I will not apologize, I love Juniper EX switches. But this meant that we had to often create a buffer between tower sites with Juniper gear and tower sites with legacy Cisco switches.

The Juniper EX3200 was the standard, if I got my way switch. Later replaced by the EX3300. The legacy switch was the Cisco 2955.  The 2955 only speaks Cisco’s pre standard MSTP implementation. The Juniper switches only spoke the industry standard. So, migrating between the two required the introduction of an intermediary. We chose the Cisco 2960s series. The 2960s will speak both, they will exchange BPDUs with “pre-standard” 2955s, as well as standard compliant, modern switches. This meant we literally had to place a 2960 in between any Juniper, and any legacy Cisco deployment. A 2955 adjacent to a Juniper switch will not communicate MSTP frames…… at best it will be reasonably RSTP sane.

This would have been fine, except……. we kept bringing 2955s back into service. As I told one vendor when they inquired about our switch life cycle….. “we run a switch until the smoke comes out, then we go out with a soldering iron, and put the smoke back in, and we run it some more!” The 2955 series hit end of life in early 2013, we were still deploying them in May 2016….. the month I left.

Reason number god only knows I left this job…… I regularly had to point out that we could not safely deploy a spare 2955 at new site X, because it would literally break the network.” I was always told I was being difficult or accused of wanting more shiny new toys.

The culmination of hell was actually managing individual links of the western core. Two MSTP instances had become four, two east, and two west. After several colossal failures due to oversubscribed links, the two western instances became nine. Yes, nine MSTP instances.

The common instance was relegated to management traffic only at first, until reaching congestion impacted switches via SSH became an issue, and management was broken up into the multiple MSTP instances. Eight of the nine customer zones were similar, they all had the same goal: Get traffic to the only way out, the headend of wifi west. The final instance was actually an emergency services related VLAN, and fortunately, had a different ultimate destination than the rest of the customer traffic. Unfortunately, they also carried an urgency, often over antiquated equipment that had no real ability to prioritize L2 data, certainly not emergency services’ traffic. But we did consider them a higher priority. To make matters worse, the emergency services were using old analog radios, and forcing them to work over IP in a way that the manufacturer considered a very bad idea. It broke all the time, and we were always blamed.

At this point, I was literally adjusting metrics on a daily basis……. live. For those of you not in the network world, spanning tree network metrics are purely layer two. This means useful tools like traceroute do not exist. Traceroute exists at layer three, a layer which is, quite frankly, not terribly difficult to diagnose. Layer two issues by contrast, are very hard to visualize, and the tools to assist you are far, far less common.

MSTP config changes are not easy, and they come in two flavors. The first is VLAN membership to instances. Changing VLAN memberships is insanely difficult, as you have to change every switch in the same MSTP domain. So, either make it all happen at once, or carefully plan how you will break the network and then put it back together in a new form. Even if you have kickass automation, this almost universally requires a 3am maintenance window.

The second is per link metric changes, one can change link costs on a single switch without technically causing a meltdown. Changing a single metric on a single instance on a single link at one end of the network could potentially send a 500+ mbit of traffic thundering in a very new direction. Any sane IT admin would still  relegate this to a 3am maintenance window. I did not have the luxury of sanity. I started my prep at 3pm everyday, and took changes live before 4:30pm.

In my last year at $weDropPackets, adjusting metrics became a daily admin task, my task. I was overworked, nothing was ever good enough, and despite my protests, the network alternated between, “just fine,” and “the way its gotta be.” We were great, we were the best WISP ever, we could do no wrong.

The final numbers speak for themselves.

  • 1000+ square miles
  • 16 major switches
  • 20+ minor switches
  • 9 MSTP instances(and the common zone)
  • one way out for 99.95% of customer traffic.

In the end, that one way out is what killed the network on a regular basis. All traffic had to reach the headend by any means necessary. A failure to balance the daily traffic on all links would result in dropped BPDU frames somewhere. The ensuing flood of traffic over a new switch path would cause more dropped BPDU frames, and a cascade failure was the end result. Network stability became a function of my ability to think on my feet, and predict the future. Starting at 3:00pm everyday, seven days a week, I loaded core network graphs, looked for the overloaded sections, and adjusted MSTP metrics…… LIVE. It was basically a question of what sections of the network were consuming the most Netflix after school that day. It changed all the time.

I did this for over a year, and I became extremely good at it. I had to be. Because when I failed to balance the daily traffic, the cascade failure happened. It would take over 50% of our paying customers offline for an hour or more, and guaranteed I would get my ass chewed out at the next staff meeting. I proposed many solutions, and yes, they all cost money, and yes, they were all rejected. We had no money, and when we did, the network rarely saw it. Yes, our primary product was our network, and it was held together with duct tape, blood, and tears.

I am ashamed to admit, those last points were my driving force for the last year I was at $weDropPackets. Everything I did was predicated on that low level of human need. I wanted to not get yelled at, that is literally all I wanted anymore. I had recently developed Meniere’s disorder, and stress was consistently triggering vertigo spells. I would wake up Monday morning, knowing I faced yet another scream out session before the senior staff, and immediately go into vertigo. In my final year, I burned through all my sick time, and nearly half my PTO. When I finally turned in my 30 days notice, friends started noticing a difference immediately. The words that guaranteed, more than anything else, that I would not reconsider leaving were from my wife. “You have been calmer, happier, and more affectionate in the last two weeks than you have been in the last two years, I feel like I got my husband back.”

No job is worth your health and happiness, not even for a funny horror story like this one.

The Lawnmower/Stereo/Bikini/Chicken/Sig Sauer/Beer Incident

I have told this story many times over the years, but never actually tried to write it down.

Framing Elements

A long time ago, before portable compute devices with more horsepower than anyone ever needed were commonplace, and long before the first Bluetooth speakers, there was a man with a dream. Or maybe it was a woman, I suppose you really need to figure out what I was wearing the day I came up with this idea, and what you think of gender identity issues, and whatever, it was me, okay? I had the dream. It wasn’t a grand dream, it will never change the world, but it was fun, and it was mine!

The dream was a riding lawnmower with a kick ass stereo system, that had wireless, and streamed MP3s from my home file server, and operated completely hands free.

The hands free part was either a design goal, or just me admitting that I had no desire to get X working on a portable LCD screen, much less pay for such a beast in 2003, back when computer hardware was far less disposable, and rarely cheap. At this point, I think I still had 486s in service at home, and at $dayJob. In fact I know I did, the original billmax.wedroppackets.net was an AMD 486 piece of shit from hell.

The lawn mower was, and is to this day, a Toro Z4200 Timecutter. A zero turn radius, 42 inch deck, gas guzzling monster, that cuts through my 1 acre property in less than an hour……. I call her Rachel.

The Stereo

So, with the design goals in mind, prepare to be amazed with my oh so awesome solution. The stereo itself was a pair of cheap speakers, requisitioned from my first dual cassette tape deck, circa 1989. The compute workhorse was a Soekris Engineering Net4501, with a MiniPCI 802.11b wireless card, and a 3.3v PCI sound card.

For those that remember such hell, the Soekris boards were notorious for not actually providing 3.3v on their single PCI slot, they were also notoriously interupt craaaazy. Finding a sound card that actually didn’t wig out at being under-powered…. and still worked under OpenBSD, took some doing. Yes folks, I was over Linux even back in those days. Linux sucks. I wish I could even remember the make and model, because I went through hell, as did my wallet to find such a beast. Trust me, it exists, it was not easy to find.

A little electrical glue provided the rest, a dc-to-dc step down converter got me the power I needed from the mower battery. My childhood as the son of a journeymen electrician has been good for a few things in my life. Operation was simple, a simple quick release wire snap provided the connectivity to the battery, it was technically possible to run the stereo without the mower running, but why would I ever do that? The modus operandi was to connect the power, go inside, grab a beer, come out, fire up Rachel.

Meanwhile, OpenBSD booted, hopped on the household wireless, mounted $fileServer0:/home/nuintari/media/tunes via NFS, read only, of course, and grabbed a playlist. From there it was just mpg123 (or was it mpg321? I forget).  Tunes soon started flying out the cheap cassette deck speakers, and yours truly would proceed to enjoy a relaxing hour or so of yard work and beer.

Rock and Roll!

Pre, The Incident

My wife is afraid of birds, royally terrified of birds. Have you ever seen how I react to spiders? Imagine that, but with birds, it is that level of terror. Actually it isn’t, my wife isn’t the bloomin’ coward I am in the face of her fears. But, she is not a fan of them to say the least.

We live in the country, or….. maybe right on the edge of the country. As country as Northwest Ohio ever gets is the point. Country enough that the neighbors raise chickens. Chickens that are mostly free to wander, and return to the hen house at the end of the day. How they weren’t all eaten by foxes, I will never know. But, they did seem to have a thing for my lilac bushes. They would wander across the street, and nest in my lilacs. My wife hated this, she’d be out in the yard, and a chicken would appear out of nowhere, and my young, young, gorgeous lady would lose her shit and run inside. I would of course, be dispatched, usually with some kind of makeshift polearm, to shoo them away.

Occasionally, I would notice them while mowing the lawn. Rachel has some oomph behind her, and if you kill the blades, and pull the deck all the way up, you can move at a solid 12+ MPH….. with the wind. Fast enough to chase chickens. Not fast enough to catch them, not that I ever wanted to, but fast enough, and loud enough, to chase them away. Also, good for a solid laugh.

We were the new couple in the neighborhood, and the farm across the street was our only real neighbor. Turns out, they had a daughter graduating high school. We were invited to the party, which we wholeheartedly accepted on the assumption that there was likely to be beer. And, I guess we should get to know the neighbors or something.

Over the course of a fine afternoon, the father approached me, and informed me that, “I see you chasing my chickens, they give you any trouble, just shoot em, they’re good eating!”

I should point out that I live NORTH of US-6….. which anyone from Ohio will recognize as the actual Mason Dixon line of demarcation between civilization and Hicksville, USA. Someone will hate this bit, but I don’t care. South of Six Hicks are a thing, and we were a solid 40 miles North of their territory, spooky.

I should note that this phenomenon exists only in Ohio. Once you reach Kentucky, the hick meter resets back to a sane level, people are way nicer, and supremely less racist. South Ohio sucks ass.

Now, I have zero interest in shooting a chicken. For starters, I own a few guns, none of them suitable for avians. Can you imagine actually hitting a chicken with a 12 gauge? Or a 7.62 SKS? It’d be feathers and a fine mist. But, even assuming I killed it, and left it intact, who wants to clean it? My old man took me hunting a few times, cleaning the carcass is the nasty part I never want to experience again.

The Incident

This part is actually pretty short, the lead up is what makes the story funny.

The stage is set, Nuintari, the man with a dream, is riding a hacked up stereo laden lawn mower, listening to classic thunder, and of course, I have a beer , and I am wearing daisy duke shorts, and a bikini top. It is either truly awesome, or truly awful to live next door to me, even if the houses are fairly far apart.

A chicken waddles over the street, through my side yard, and right into my lilac bush.

It should be noted that at this time in my life, I had come into possession of two key items relevant to this story. A Sig Sauer, P229 9mm handgun, and a pile of 9mm blanks. Remember, I don’t actually want to kill the chicken, I just wanna fuck with it. Also, I am drinking.

I know, I know, I know, I shouldn’t mix beer and guns….. It hasn’t happened since…… that I can recall.

So, naturally, inside I go, grab the gun, a fresh beer (I know, I know), and load the weapon with blanks. Upon returning to Rachel, the stereo is now beginning to play Wagner’s Ride of The Valkyries. It was so on. Deck up, blades off, LETS GET THOSE CHICKENS!

The next few minutes is basically me, in a Bikini top, daisy duke shorts, driving a zero turn radius mower, with a beer in one hand, a blanks loaded 9mm handgun in the other, rocking out to classical German musical great Wagner, chasing a chicken around my yard, occasionally taking potshots at it with the blanks…… and of course, laughing like an idiot the entire time.

At one point, I caught a look from the farmer across the street, who was basically, as the kids say, “losing his shit.”

The Legacy

The stereo blew up. A victim of a replacement battery, and operator failure to observe reversed poles….. oops. It has since been replaced with a smart phone, a bluetooth headset, and Pandora. Not as sexy, but it works. The neighbor moved away, the chickens are all gone, the farm is largely empty these days, some days, I can chase a Killdeer around a bit, but it just isn’t the same. Killdeer fight back.