Juniper: VPLS Multihoming - Multiple PE

There's actually a few ways to avoid bridging loops in a VPLS network. Bridging in a VPLS environment is not really different from a standard Ethernet network, a spanning-tree protocol like the original IEEE 802.1D or any of its variants like RSTP or MSTP can be enabled to block the redundant link(s). Ethernet Ring Protection (ERP) could also be enabled on platforms supporting it (e,g. Juniper MX series), but certainly the most common and effective way is to carefully provision the VPLS VPN instances using BGP and to respect a few basic rules.

For the sake of testing VPLS multihoming with multiple PE on Junos, I built a rather small lab, using two VMware ESXi hypervisors and a bunch of Juniper M-series routers. The routers are physically connected to the hardware nodes where the hypervisors are residing, and the latter use PCI Passthrough (DirectPath I/O in VMware terminology) to transparently connect the nodes' NICs to the guest VMs. By doing so, I remove the VMware software switches from my tests, which's somewhat of a very good idea as my experiments require direct connectivity between my PE and CE devices.

Here's the scenario. A company buy VPLS service to connect at Layer-2 their two distant corporate networks. One customer site is in Geneva and the other is in Amsterdam. The former site in Geneva has a connectivity to a Provider Edge (PE) router in Milan, and a second, redundant connectivity to a PE in the same city (I know this sound odd, but the second link became available later in my mind). The CE devices here are functionally equivalent to ethernet switches as the VMs are using the standard linux kernel bridge. The question is, what happen if we provision each VPN sites, without even considering the potential bridging loop introduced by the redundant link in Geneva ?

To answer that question, I quickly provisioned each sites independently from each other on the Milan, Geneva and Amsterdam PE. Please note, that I always use the router-id of my PE for the administrative part of my route distinguisher, so I can more easily identify from which PE a VPN route belong to. The CE site number '1' and '3' is in fact the same CE in Geneva; a debian guest with two distinct NICs.

root@lab-m7i-1> show vpls connections logical-system Milan    
Layer-2 VPN connections:

Legend for connection status (St)   
EI -- encapsulation invalid      NC -- interface encapsulation not CCC/TCC/VPLS
EM -- encapsulation mismatch     WE -- interface and instance encaps not same
VC-Dn -- Virtual circuit down    NP -- interface hardware not present 
CM -- control-word mismatch      -> -- only outbound connection is up
CN -- circuit not provisioned    <- -- only inbound connection is up
OR -- out of range               Up -- operational
OL -- no outgoing label          Dn -- down                      
LD -- local site signaled down   CF -- call admission control failure      
RD -- remote site signaled down  SC -- local and remote site ID collision
LN -- local site not designated  LM -- local site ID not minimum designated
RN -- remote site not designated RM -- remote site ID not minimum designated
XX -- unknown connection status  IL -- no incoming label
MM -- MTU mismatch               MI -- Mesh-Group ID not available
BK -- Backup connection          ST -- Standby connection
PF -- Profile parse failure      PB -- Profile busy
RS -- remote site standby        SN -- Static Neighbor
VM -- VLAN ID mismatch

Legend for interface status 
Up -- operational           
Dn -- down

Instance: vpls-1
  Local site: 1 (1)
    connection-site           Type  St     Time last up          # Up trans
    2                         rmt   Up     Jan  1 01:43:48 1999           1
      Remote PE: 10.0.255.6, Negotiated control-word: No
      Incoming label: 800001, Outgoing label: 800000
      Local interface: vt-1/2/0.84935169, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls vpls-1 local site 1 remote site 2
    3                         rmt   Up     Jan  1 01:37:58 1999           1
      Remote PE: 10.0.255.2, Negotiated control-word: No
      Incoming label: 800002, Outgoing label: 800008
      Local interface: vt-1/2/0.84935168, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls vpls-1 local site 1 remote site 3

root@lab-m7i-2> show vpls connections logical-system Geneva    
Layer-2 VPN connections:

[...]

Instance: vpls-1
  Local site: 3 (3)
    connection-site           Type  St     Time last up          # Up trans
    1                         rmt   Up     Jul 12 13:38:13 2013           1
      Remote PE: 10.0.255.12, Negotiated control-word: No
      Incoming label: 800008, Outgoing label: 800002
      Local interface: vt-0/3/0.51380480, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls vpls-1 local site 3 remote site 1
    2                         rmt   Up     Jul 12 13:44:03 2013           1
      Remote PE: 10.0.255.6, Negotiated control-word: No
      Incoming label: 800009, Outgoing label: 800002
      Local interface: vt-1/2/0.51380481, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls vpls-1 local site 3 remote site 2

root@lab-m7i-2> show vpls connections logical-system Amsterdam  
Layer-2 VPN connections:

[...]

Instance: vpls-1
  Local site: 2 (2)
    connection-site           Type  St     Time last up          # Up trans
    1                         rmt   Up     Jul 12 13:44:03 2013           1
      Remote PE: 10.0.255.12, Negotiated control-word: No
      Incoming label: 800000, Outgoing label: 800001
      Local interface: vt-1/2/0.17826048, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls vpls-1 local site 2 remote site 1
    3                         rmt   Up     Jul 12 13:44:03 2013           1
      Remote PE: 10.0.255.2, Negotiated control-word: No
      Incoming label: 800002, Outgoing label: 800009
      Local interface: vt-0/3/0.17826049, Status: Up, Encapsulation: VPLS
        Description: Intf - vpls vpls-1 local site 2 remote site 3

All VPLS connections are 'UP' and running. I can now configure the first CE device in Geneva identified as 'lab-debian-01'.

root@lab-debian-01# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST>  mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether a0:36:9f:18:38:90 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST>  mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether a0:36:9f:18:38:91 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 00:50:56:81:46:f8 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST>  mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 00:50:56:81:60:fa brd ff:ff:ff:ff:ff:ff

The VM has four ethernet interfaces. eth2 is used as a out-of-band management interface for the lab and is actually the only interface in the UP state. eth0 and eth1 are respectively connected to the Milan PE 'fe-0/0/1' and Geneva PE 'fe-0/0/2' interfaces. Now it's time to configure the customer's bridge to demonstrate how much the loop can be devastating.

root@lab-debian-01# brctl addbr vlan900
root@lab-debian-01# vconfig add eth0 90
root@lab-debian-01# vconfig add eth1 900
root@lab-debian-01# ip link set eth0 up
root@lab-debian-01# ip link set eth0.900 up
root@lab-debian-01# ip link set eth1 up
root@lab-debian-01# ip link set eth1.900 up
root@lab-debian-01# brctl addif vlan900 eth0.900
root@lab-debian-01# brctl addif vlan900 eth1.900
root@lab-debian-01# ip link set vlan900 up
root@lab-debian-01# brctl show
bridge name	bridge id		STP enabled	interfaces
vlan900		8000.a0369f183890	no		eth0.900
							eth1.900

The first signs of a bridging loop came rapidly. The customer CE device was already complaining!

[ 3643.537853] vlan900: received packet on eth0.900 with own address as source address
[ 3643.537864] vlan900: received packet on eth0.900 with own address as source address
[ 3643.537876] vlan900: received packet on eth0.900 with own address as source address
[ 3643.537884] vlan900: received packet on eth0.900 with own address as source address
[ 3643.537890] vlan900: received packet on eth0.900 with own address as source address
[ 3643.537903] vlan900: received packet on eth1.900 with own address as source address
[ 3643.537909] vlan900: received packet on eth1.900 with own address as source address
[ 3643.537915] vlan900: received packet on eth1.900 with own address as source address
[ 3643.537922] vlan900: received packet on eth1.900 with own address as source address
[ 3643.537928] vlan900: received packet on eth1.900 with own address as source address
[ 3648.534105] net_ratelimit: 204137 callbacks suppressed

And the Geneva PE router was already looping around 30Mbps of traffic:

root@lab-m7i-2> show interfaces fe-0/0/2 | match rate 
  Input rate     : 31344424 bps (53057 pps)
  Output rate    : 29830032 bps (50711 pps)

This is the expected behavior. After all, how our PE routers could possibly know there's a loop? Another sign of a routing loop is the constant flapping of the MAC addresses from one PE to the other.

root@lab-m7i-2> show route forwarding-table family vpls | find Geneva 
Logical system: Geneva
Routing table: vpls-1.vpls
VPLS:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct  1226     1
fe-0/0/2.900       user     0                    comp  1237     2
vt-0/3/0.51380480  user     0                    comp  1208     3
vt-1/2/0.51380481  user     0                    comp  1208     3
a0:36:9f:18:36:64/48 dynm     0                  indr 262151     5
                              10.0.0.18         Push 800002   724     2 fe-1/3/0.604
a0:36:9f:18:38:90/48 dynm     0                  indr 262151     5
                              10.0.0.18         Push 800002   724     2 fe-1/3/0.604

root@lab-m7i-2> show route forwarding-table family vpls | find Geneva    
Logical system: Geneva
Routing table: vpls-1.vpls
VPLS:
Destination        Type RtRef Next hop           Type Index NhRef Netif
default            perm     0                    rjct  1226     1
fe-0/0/2.900       user     0                    comp  1237     2
vt-0/3/0.51380480  user     0                    comp  1208     3
vt-1/2/0.51380481  user     0                    comp  1208     3
a0:36:9f:18:36:64/48 dynm     0                  ucst   722     4 fe-0/0/2.900
a0:36:9f:18:38:90/48 dynm     0                  ucst   722     4 fe-0/0/2.900

As the output above show, the MAC address of the Geneva CE's bridge (a0:36:9f:18:38:90) is once considered remote (see push operation), and once considered local.

To prevent the loop from occurring between multiple PE, the easiest way is to configure the redundant PE instances using the same route distinguisher, the same site identifier and to activate the multi-homing knob under the VPLS site configuration, as follow:

Geneva PE
routing-instances {
    vpls-1 {
        instance-type vpls;
        interface fe-0/0/2.900;
        route-distinguisher 10.0.255.12:100;
        vrf-target target:57413:100;
        protocols {
            vpls {
                site-range 8;
                site 1 {
                    site-identifier 1;
                    multi-homing;
                    site-preference 200;
                    interface fe-0/0/2.900;
                }
            }
        }
    }
}

Milan PE
routing-instances {
    vpls-1 {
        instance-type vpls;
        interface fe-0/0/1.900;
        route-distinguisher 10.0.255.12:100;
        vrf-target target:57413:100;
        protocols {
            vpls {
                site-range 8;
                site 1 {
                    site-identifier 1;
                    multi-homing;
                    site-preference 100;
                    interface fe-0/0/1.900;
                }
            }
        }
    }
}

I chose here to use the router distinguisher of the first provisioned PE, namely Milan. I also used the 'site-preference' knob to prefer one site over another. By default, all multi-homed site has a site-preference of 100. As the local-preference BGP attribute, the higher the value, the more preferred is the route.

After the commit operation, the Milan PE's VPLS connection state changed, and the PE-CE link was marked as 'LN' for 'local site not designated'. It means the PE actually know there's a redundant link, and it choose not to advertise it anymore, until... well, the other PE's link fail.

root@lab-m7i-1> show vpls connections logical-system Milan     
Layer-2 VPN connections:

Legend for connection status (St)   
EI -- encapsulation invalid      NC -- interface encapsulation not CCC/TCC/VPLS
EM -- encapsulation mismatch     WE -- interface and instance encaps not same
VC-Dn -- Virtual circuit down    NP -- interface hardware not present 
CM -- control-word mismatch      -> -- only outbound connection is up
CN -- circuit not provisioned    <- -- only inbound connection is up
OR -- out of range               Up -- operational
OL -- no outgoing label          Dn -- down                      
LD -- local site signaled down   CF -- call admission control failure      
RD -- remote site signaled down  SC -- local and remote site ID collision
LN -- local site not designated  LM -- local site ID not minimum designated
RN -- remote site not designated RM -- remote site ID not minimum designated
XX -- unknown connection status  IL -- no incoming label
MM -- MTU mismatch               MI -- Mesh-Group ID not available
BK -- Backup connection          ST -- Standby connection
PF -- Profile parse failure      PB -- Profile busy
RS -- remote site standby        SN -- Static Neighbor
VM -- VLAN ID mismatch

Legend for interface status 
Up -- operational           
Dn -- down

Instance: vpls-1
  Local site: 1 (1)
    connection-site           Type  St     Time last up          # Up trans
    1                         rmt   LN   
    2                         rmt   LN   

The Milan PE's RIB shown three routes, two were advertised from the backbone's route-reflectors and originated from Geneva, and the last one, was the local route from Milan, which's was not advertised to BGP anymore.

root@lab-m7i-1> show route logical-system Milan table vpls-1.l2vpn.0 extensive | find 10.0.255.12:100
 10.0.255.12:100:1:1/96 (3 entries, 1 announced)
        *BGP    Preference: 170/-201
                Route Distinguisher: 10.0.255.12:100
                Next hop type: Indirect
                Address: 0x9242c80
                Next-hop reference count: 5
                Source: 10.0.255.1
                Protocol next hop: 10.0.255.2
                Indirect next hop: 2 no-forward
                State: <Secondary Active Int Ext>
                Local AS: 57413 Peer AS: 57413
                Age: 15:38      Metric2: 1 
                Task: BGP_57413.10.0.255.1+64030
                Announcement bits (1): 0-vpls-1-l2vpn 
                AS path: I (Originator) Cluster list:  1.1.1.1
                AS path:  Originator ID: 10.0.255.2
                Communities: target:57413:100 Layer2-info: encaps:VPLS, control flags:, mtu: 0, site preference: 200
                Import Accepted         
                Label-base: 800016, range: 8
                Localpref: 200
                Router ID: 10.0.255.1
                Primary Routing Table bgp.l2vpn.0
                Indirect next hops: 1
                        Protocol next hop: 10.0.255.2 Metric: 1
                        Indirect next hop: 2 no-forward
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 10.0.0.17 via ge-1/3/0.604
                        10.0.255.2/32 Originating RIB: inet.3
                          Metric: 1                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 10.0.0.17 via ge-1/3/0.604
         BGP    Preference: 170/-201
                Route Distinguisher: 10.0.255.12:100
                Next hop type: Indirect
                Address: 0x9242c80
                Next-hop reference count: 5
                Source: 10.0.255.7
                Protocol next hop: 10.0.255.2
                Indirect next hop: 2 no-forward
                State: <Secondary NotBest Int Ext>
                Inactive reason: Not Best in its group - No difference
                Local AS: 57413 Peer AS: 57413
                Age: 15:38      Metric2: 1 
                Task: BGP_57413.10.0.255.7+57023
                AS path: I (Originator) Cluster list:  1.1.1.1
                AS path:  Originator ID: 10.0.255.2
                Communities: target:57413:100 Layer2-info: encaps:VPLS, control flags:, mtu: 0, site preference: 200
                Import Accepted
                Label-base: 800016, range: 8
                Localpref: 200
                Router ID: 10.0.255.7
                Primary Routing Table bgp.l2vpn.0
                Indirect next hops: 1
                        Protocol next hop: 10.0.255.2 Metric: 1
                        Indirect next hop: 2 no-forward
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 10.0.0.17 via ge-1/3/0.604
                        10.0.255.2/32 Originating RIB: inet.3
                          Metric: 1                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 10.0.0.17 via ge-1/3/0.604
         L2VPN  Preference: 170/-101
                Next hop type: Indirect
                Address: 0x92400e8
                Next-hop reference count: 2
                Protocol next hop: (null)
                Indirect next hop: 0 -
                State: <Int Ext>
                Inactive reason: Route Preference
                Age: 2:12:29    Metric2: 1 
                Task: vpls-1-l2vpn
                AS path: I
                Communities: Layer2-info: encaps:VPLS, control flags:, mtu: 0, site preference: 100
                Label-base: 800000, range: 8, status-vector: 0x7F 

This conclude this post on VPLS multihoming with multiple PE on Juniper Junos.

About the author Nicolas Chabbey

Nicolas Chabbey is a Network Engineer certified with Cisco Systems and Juniper Networks. He has begun his career in 2003, and has designed, implemented and maintained networks for enterprises and service providers. When he is not behind a computer, he is riding his mountain bike across the Swiss alps.

Previous Home Next

Comments

blog comments powered by Disqus