VyOS and Redundancy

Some time ago, perhaps three years at this point, I had to swallow my pride and switch from AT&T DSL back to Comcast Cable internet. When I did so, and got a sizable boost in bandwidth to go along with it, I opted to change my internal networking from using my wireless router as the edge device behind the modem and go with something more functionally complete for the task. The end result of that decision led me to make use of an appliance-style router based on VyOS. I would be able to use it a virtual machine, which was where my home lab was leading to, and I’d be able to gain all the benefits of my new bandwidth. However, one thing bothered me in that, though I had multiple hypervisors to run VMs on, my network access was tethered to one piece of hardware.

I recently got motivated to fix all that, and thus, I have finally solved that problem by building out redundant devices and linking them together using VyOS’s implementation of the Virtual Router Redundancy Protocol (VRRP for short). I had attempted to use Clustering as available in VyOS to do the same task, but common problems such as ARP caching and induced latency produced undesirable behavior when the active router failed over. VRRP solved much of that, though my walk to the finish line was not without complications.

The Setup

Configuring VyOS for VRRP isn’t a difficult chore. It is actually fairly well documented, even if it isn’t well explained. The configuration for VRRP is built directly on the interface itself and not in one of the other configuration sections available. Quoting the VyOS Wiki, the configuration options are as follows for version 1.1.8 (current as of this writing).

interfaces ethernet eth0 vrrp 
   vrrp-group <1-255> #VRRP group number
      advertise-interval <1-255> #Advertise interval (default 1)
      authentication
      description  #Description
      disable  #VRRP group disabled
      hello-source-address <x.x.x.x> #Source address for vrrp hello packets (optional)
      preempt <boolean> #Preempt mode
      preempt-delay <0-1000> #Preempt Delay in seconds
      priority <1-255> #Priority
      rfc3768-compatibility
      run-transition-scripts  #Scripts for VRRP state-transitions
      sync-group <text> #Add this vrrp group to a sync group
      virtual-address <x.x.x.x> #Virtual IP address (up to 20 per group)

There are some required options in the above configuration section, priority and virtual-address being the bulk of them. Preempt is set to True by default, so on a passive node, you have to set it to be false. Priority can also be different, and probably should be between different members of the group, but the VRRP protocol contains a way to sort out the election of a new master even if the priorities are the same.

One option I was highly interested in, however, was the rfc3768-compatibility. This instructs the router to create a virtual interface so that it can assign a well-defined MAC address to it, one that will float with whichever router is master in the group. This virtual interface will also carry the virtual-address, and will not be available on the passive nodes. It will be important for firewall rules to take into account this new interface, however, as traffic may be routed over it.

In my case, I set up my primary interface with the following settings.

interfaces {
  ethernet eth0 {
    address 192.168.0.2/24
    description "Internal Network"
    duplex auto
    smp_affinity auto
    speed auto
    vrrp {
      vrrp-group 10 {
        advertise-interval 1
        authentication {
          password DEADBEEF
          type ah
        }
        preempt false
        priority 100
        rfc3768-compatibility
        sync-group RouterCore
        virtual-address 192.168.0.1/24
      }
    }
  }
}

With the passive node, the address set is 192.168.0.3/24 instead. I have since found out that this may be suboptimal as traffic doesn’t always flow exactly as I expect it to. For the most part, it acts fine, but because of the NAT I’m doing going out the WAN interface, and the fact the traffic is flowing out using the router’s unique IP address instead of the virtual address, this causes problems with connections during failover.

This got me started but complications started getting in the way. Those issues and their resolution will be part of Part 2 of this series.