Home Network of Doom Part II: VPLS for Maniacs

Now that you have your new mess of tangled wires, and miles of configuration out of the way, it is time to start firing up some useful services. My entire initial reason for learning about MPLS was to get VPLS. As a consequence, this entire series of articles revolves around building a giant, virtual switch that spans multiple L3 devices. This particular article will focus on getting basic VPLS working. I will go into filtering, load balancing, and other such complex topics at a later time. Conceptually, the end result of this article can be compared to a VLAN aware switch nestled comfortably on top of an existing IP network. Why this is awesome is that, once built, you have all the advantages of an Ethernet network delivered to the edge, with the ability to use layer 3 tools and techniques for management of the core.

Warning: SRX Limitations

When delivering Ethernet services end to end, a service provider would(should) concern themselves with loop detection. The most common way to achieve this would be to run rapid spanning tree protocol (RSTP). Another key piece would be dropping inbound BPDU frames from customer devices, or disabling the port entirely when BPDU frames are detected. This is done to prevent a customer from joining your spanning tree network, as spanning tree has no security options. RSTP over VPLS is not supported by the branch SRX platform, nor can you configure bpdu-block in any useful way for VPLS.

What you can do is configure a firewall rule to act as a basic BPDU Filter.

nuintari@garage-mpls4> show configuration firewall  
family vpls {
    filter flood-ctrl {
        term stp {
            from {
                destination-mac-address {
                    01:80:c2:00:00:00/48;
                }
            }
            then {
                count stp;
                discard;
            }
        }
        term a-stp {
            from {
                destination-mac-address {
                    01:80:c2:00:00:00/44;
                }
            }
            then {
                count a-stp;
                discard;
            }
        }
        term pvst {
            from {
                destination-mac-address {
                    01:00:0c:cc:cc:cd/48;
                }
            }
            then {
                count pvst;
                discard;
            }
        }
        term cdp {
            from {
                destination-mac-address {
                    01:00:0c:cc:cc:cc/48;
                }
            }
            then {
                count cdp;
                discard;
            }
        }
        term vlan-br {
            from {
                destination-mac-address {
                    01:00:0c:cd:cd:ce/48;
                }
            }
            then {
                count vlan-br;
                discard;
            }
        }
        term stp-upfast {
            from {
                destination-mac-address {
                    01:00:0c:cd:cd:cd/48;
                }
            }
            then {
                count stp-upfast;
                discard;
            }
        }
        term default {
            then accept;
        }
    }
}

Special thanks to Mike for this tip. I have included the set statements for this firewall here.

Simply apply it to a VPLS routing instance.

set routing-instances <instance-name> forwarding-options family vpls flood input flood-ctrl

With a little JunOS scripting, one could conceivably turn this into a form of BPDU Guard as well. Sadly, this does not address loop detection and protection. The other method to protect against loops would be to configure mac move protections, which again, is not supported by the Branch SRX platform. In short, don’t create any loops!

The Basic Layout

To the outside observer, my entire network behaves like a giant VLAN aware switch. I’ll demonstrate my configurations for a few devices spanning three of my more important VLANs. The whole thing looks something like this:

VLAN 1000 is my general purpose, internet access network. My workstations live here, the wifi access lives here, my various gadgets including my Roku and Chromecast live here. If I had a single LAN home like most people, it would be this network.

But since I am insane, I also maintain a network for servers on VLAN 1005. Okay, I actually maintain several, but VLAN 1005 is for all the basic services that the rest of the network needs to exist, such as DNS and DHCP.

I also maintain shared storage for servers, and my one UNIX desktop over a third VLAN, 1009. Basically a network of NFS and iSCSI traffic. This network is not handed up to the IP router, as there is no need.

I maintain about a dozen other VLANs, but for simplicity sake, I’ll be cutting them out of configurations, hopefully I don’t miss anything.

Before I even get into the config, yes, my home network is named ‘assylum,’ I name all my computers after mental disorders. Also yes, I am aware that it is not the correct spelling for asylum, it is a joke, try not to think too hard to get it. My proofreader mother thought it was hilarious.

Interface Configuration

The configurations for this entire mess are substantially shorter than the previous post, and I won’t be displaying each and every device’s configuration this time around. Interface examples will start with the IP router, and a few examples of PE interfaces.

For clarity sake, lets take a look at my IP Gateway’s configuration.

nuintari@headend-mpls3> show configuration interfaces ge-0/0/1
description core-mpls5:ge-0/0/0;
vlan-tagging;
unit 1000 {
    description assylum-nat-1000;
    vlan-id 1000;
    family inet {
        no-redirects;
        address 192.168.119.254/24 {
            primary;
            preferred;
        }
    }
}
unit 1005 {
    description assylum-srv-1005;
    vlan-id 1005;
    family inet {
        no-redirects;
        address 192.168.250.254/24 {
            primary;
            preferred;
        }
    }
}

Nothing special, just VLANs and IP. The router effectively acts as a CE device in MPLS parlance. I’ll skip any real explanation here, and assume you know how a router basically works and is configured.

Exactly opposite on that same wire is core-mpls5, the, “entrance” to my VPLS switching layer.

description headend-mpls3:ge-0/0/1;
flexible-vlan-tagging;
mtu 1624;
encapsulation vlan-vpls;
unit 1000 {
    description assylum-nat-1000;
    encapsulation vlan-vpls;
    vlan-id 1000;
    family vpls;
}
unit 1005 {
    description assylum-srv-1005;
    encapsulation vlan-vpls;
    vlan-id 1005;
    family vpls;
}

This is performing the role of a PE device in MPLS-ese. The only real thing you need to take in, is that we are declaring an encapsulation of vlan-vpls at both the interface, and the sub-interface level, as well as declaring family vpls on each sub-interface. This is all required, on platforms with greater interface flexibility such as the MX, you can have different types of sub-interfaces on the same physical port. Sadly, the Branch SRX platform isn’t that awesome. We have also declared flexible-vlan-tagging for the interface itself, which becomes more important later. This port, as it stands right now, could operate with just “vlan-tagging” declared on the interface, but that would constrain us later on, which I will touch on in a bit. In short, if you don’t need any particular configuration for any particular reason, stick with flexible, because it is just that, flexible.

For purposes of wrapping your head around it, think of it like a switch performing as a trunk port. For a counter-example, here is an EX2200 switch passing the same VLANs (plus my storage network) in a pure L2 setting:

nuintari@core-sw0> show configuration interfaces ge-0/0/0
description core-mpls5:ge-0/0/1;
unit 0 {
    family ethernet-switching {
        port-mode trunk;
        vlan {
            members [ assylum-nat-1000 assylum-srv-1005 assylum-nfs-1009 ];
        }
    }
}

Directly opposite that port on core-mpls5 looks like this:

nuintari@core-mpls5> show configuration interfaces
ge-0/0/2 {
    description core-sw0:ge-0/0/0;
    flexible-vlan-tagging;
    mtu 1624;
    encapsulation vlan-vpls;
    unit 1000 {
        description assylum-nat-1000;
        encapsulation vlan-vpls;
        vlan-id 1000;
        family vpls;
    }
    unit 1005 {
        description assylum-srv-1005;
        encapsulation vlan-vpls;
        vlan-id 1005;
        family vpls;
    }
    unit 1009 {
        description assylum-nfs-1009;
        encapsulation vlan-vpls;
        vlan-id 1009;
        family vpls;
    }
}

Pretty standard config so far, but what happens if you want to present an untagged ethernet frame directly on the PE device? My wireless access points are dumb, they don’t understand VLANs (but they have killer antenna), so I need to somehow configure a VPLS speaking interface to act like an access port on a VLAN aware switch. Question is, how do I do that?

The short answer is, you don’t. Asking VPLS to act exactly like a VLAN aware switch belies their true nature in that they aren’t exactly mirror clones. With that in mind, you can fake an access port reasonably well.

nuintari@garage-mpls4> show configuration interfaces
ge-0/0/2 {
    description assylum-wifi;
    flexible-vlan-tagging;
    native-vlan-id 1000;
    mtu 1624;
    encapsulation vlan-vpls;
    unit 1000 {
        description assylum-nat-1000;
        encapsulation vlan-vpls;
        vlan-id 1000;
        family vpls;
    }
}

In reality, this is no different than any other VPLS speaking interface, except we have added the ‘native-vlan-id’ statement.  There is another piece to this puzzle which we will cover in a bit, but what you are now performing is called ‘VLAN normalization’, and only works on interfaces in ‘flexible-vlan-tagging’ mode. The other important piece is in the routing instance configuration. This particular trick only works in VPLS instances where we declare a single VLAN. Fortunately, you can do this on a per case basis. In most provider networks, it would be most advantageous to provide a simple tube, whatever the customer puts on the wire, tags or no tags, comes out the other end the same. If you want to be play fake access port, you lose this ability. I think, this might be way wrong, I wrote this article, stopped fact checking, and posted it a month and a half later……

Routing Instances

The actual work of gluing the interfaces together across the network falls to the routing instances. There are three and a half major pieces that require a bit of explanation. The route-distinguisher, vrf-target, and site-range and site-identifier. The rest of the configuration is fairly self explanatory, this is the routing instance for vlan 1000, my general purpose network.

nuintari@garage-mpls4> show configuration routing-instances
assylum-nat-1000 {
    instance-type vpls;
    vlan-id 1000;
    interface ge-0/0/1.1000;
    interface ge-0/0/2.1000;
    route-distinguisher 10.0.7.4:1000;
    vrf-target target:65001:1000;
    forwarding-options {
        family vpls {
            flood {
                input flood-ctrl;
            }
        }
    }
    protocols {
        vpls {
            site-range 10;
            no-tunnel-services;
            site garage-mpls4:assylum-nat-1000:pe4 {
                site-identifier 4;
            }
        }
    }
}

The vrf-target is essentially, the identifier for the l2vpn. Each device taking part in the same VPLS VRF should have the same vrf-target. They take the form of target:<asn>:<uid>. Since I have been creating single vlan pseudowires, I have taken to using the VLAN I am flinging around as the identifier, but as single VLANs are not all VPLS can do, this is hardly a requirement. The ASN I used from the previous article in this series was 65001, so we use that here as well.

The route-distinguisher takes the form of <any ipv4 addr>:<uid>, but the common convention is to use this router’s loopback address, and again, I use the VLAN I am moving around on the pseudowire for the uid. The best way I can explain the difference between vrf-target and the route-distinguisher is, the vrf-target defines a VPLS instance, network wide, the route-distinguisher helps identify the individual members. With all analogies, there is more to it than that, but its a pretty small piece of VPLS. The RD matters much more when you are talking about MPLS l3vpn, and moving the same address space for disparate customers. If you elected to forego BGP signaling, and are doing all your signaling with LDP, this option is completely unnecessary.

The final pairing, site-range and site-identifier, serve to identify the individual members of the VPLS VRF internally, as well as dictate if and when pseudowires are formed. The site-range dictates the highest site-identifier that this instance will form a pseudowire against…… What? Okay, there are three common scenarios where this either really matters, or means next to nothing.

The most obvious is a point to point connection. With only two members, the site-range could be 2, with site-identifiers 1 and 2. Nice and self documenting, this VRF is a tube, what goes in one end comes out the other.

The next is the full mesh, where these settings matter the least. In a scenario where full pseudowires  between all sites are desired, one could opt to accept the default site-range (its 65,534) and use automatic-site-id in lieu of outright declaring anything. For the purposes of actually learning anything, I like to adjust all these options, lest I never figure out why you would ever want to.

Where this really matters is when a complex topology of hubs and spokes is the desired behavior. Edge devices that only connect to one or two devices in the core should not be wasting resources forming pseudowires across the network. I don’t own enough hardware to build this scenario(yet), but when I do, I’ll post some examples.

So this all being said, I standardized on a site-range of 10 for everything except the instance that carries my internet traffic. My cable modem sits in my family room near the TV, and is carried over my VPLS network to my core router in the server room, so a true point to point link, where I used a site-range of 2.  With that one exception in mind, I used my loopback addresses to determine site-identifiers. This won’t scale far, but helps keep me sane at home.

Diagnostics and a Rosetta Stone

Going along with our analogy that this is just one big VLAN aware switch, we will need some basic commands to verify the insanity we have created.

First of all, we want to see if our pseudowires have actually come up, which is pretty easy:

show vpls connections

Yes, for some reason, this one singular JunOS command comes with a built in legend. Easily ignored over SSH, really friggin annoying at 9600 bauds of serial cable love…… The most important thing to look for here is Remote PE: entries. A mis-configured instance, or a bifurcated network will instead show entries with things like “No connections found.”

Good:

Instance: assylum-nat-1000
Edge protection: Not-Primary
  Local site: core-mpls5:assylum-nat-1000:pe0 (5)
    connection-site Type St Time last up # Up trans
    1 rmt Up Mar 27 17:49:01 2017 1
      Remote PE: 10.0.7.1, Negotiated control-word: No
      Incoming label: 262153, Outgoing label: 262149
      Local interface: lsi.1048576, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls assylum-nat-1000 local site 5 remote site 1
   2 rmt Up Mar 27 17:49:09 2017 1
     Remote PE: 10.0.7.2, Negotiated control-word: No
     Incoming label: 262154, Outgoing label: 262157
     Local interface: lsi.1048578, Status: Up, Encapsulation: VPLS
       Description: Intf - vpls assylum-nat-1000 local site 5 remote site 2
  4 rmt Up Mar 29 11:23:53 2017 1
    Remote PE: 10.0.7.4, Negotiated control-word: No
    Incoming label: 262156, Outgoing label: 262157
    Local interface: lsi.1048586, Status: Up, Encapsulation: VPLS
      Description: Intf - vpls assylum-nat-1000 local site 5 remote site 4

Bad:

Instance: assylum-sdc-1006
Edge protection: Not-Primary
  Local site: core-mpls5:assylum-sdc-1006:pe5 (5)
  Local site: core-mpls5:assylum-sdc-1006:pe10 (10)
No connections found.

How fortuitous! A VLAN for a dead project I have long since forgotten about, but have failed to rip out of the network layer! This also shows that a local instance can be more than a single site, more on that later….

The next thing you might care about is whether or not these instances are actually moving any real traffic, we have a command for that:

show vpls statistics

Basic counters, bits in, bits out, zeros probably mean bits are not moving. If I get to this point and get zeros from this command, it invariably means I have a typo somewhere, usually a mismatched VLAN.

Since this mess is effectively a giant, virtualized switch, you will probably want to view the MAC address table. If this was a pure L2 environment, this would suffice:

show ethernet-switching table

Sadly, this gets us nothing in a VPLS enviroment. Instead, you need to type out this handful:

show route forwarding-table family vpls

If you all you care about is MAC address entries, be prepared to visually filter a bit. Sometimes you just want to know the answer to a simple question like, “DO YOU FRIGGIN SEE THIS MAC?!?!?!” Plenty of other, occasionally very useful, information is presented here. This is another fine example of something that doesn’t paste well, so here is a text dump.

Some MAC entries have next-hops listed, others do not. In case you haven’t figured it out, this tells you if the MAC is directly attached, or comes form somewhere else in the VPLS network. Either way, it is kind enough to tell you which interface it came from. This is a fine time to yank a cable out, assuming you constructed a redundant path, and re-check the output of this command. If you have a mesh or any redundant sort, MACs will move, otherwise, they will time out and disappear.

Moof

I hate writing wrap ups. We have gone down the rabbit hole on MPLS/VPLS a bit further, and have something that is actually usable. In the next piece in this series, we will talk about load balancing, fault tolerance, and using L3 to troubleshoot L2, because directly troubleshooting L2 sucks, believe me I know. *mental note, write post on that subject, link it here (I’ll never remember that second part).*