Latest Posts

Archive

BGP Session Culling - BCP214

Maintenance at IXPs is a regular occurrence. Software on switches needs updating, new features are released, and relentless traffic growth means hardware must be swapped. For some years we have used a technique as part of these works which reduces the impact on the Internet.

What is an IXP?

An IXP is a specialised layer-two switch fabric enabling networks to interconnect. In its simplest form an IXP can be a single switch; as of 2018 a mid-sized IXP such as LONAP comprises around 25 switches interconnected at multiples of 100G.

Network operators connect their routers to the fabric, establishing BGP sessions and exchange traffic. BGP is the control plane, signalling reachability.

What happens during a maintenance?

A few times a year, service impacting maintenance is required. This can be anything from switch software upgrades or replacements, fibre repatches, to complete POP moves.

At LONAP we typically notify affected members by email one week in advance, and carry out the work during low traffic times (generally after 2300 localtime). In theory this gives members time to move traffic away so it is unaffected. However, operational experience shows this is a rare occurrence and most networks simply rely on waiting for BGP timers to remove traffic.

Why is this harmful?

This practice usually results in some minutes of blackholed (lost) traffic because, until the BGP session is torn down, user traffic continues to be sent across the now faulty path.

In practice, once traffic stops flowing, it takes a minute or two for one or both ends of the connection to notice that reachability has ceased, tear down the BGP sessions and begin to send the traffic elsewhere.

During this period, users’ traffic is blackholed. This “break-before-make” period is harmful, but for many years was accepted by IXPs and network operators as an unavoidable consequence of maintenance activity.

Improving the experience

It’s clear that we must practice “make-before-break”, removing affected traffic from the IXP fabric at the start of the maintenance window. This can be accomplished by the IXP operator applying ACLs (access control lists) to member ports, which cause BGP sessions to be torn down, and traffic to be removed in advance of the service affecting maintenance.

The process

At the start of the maintenance window, cull the BGP sessions
Wait for traffic to diminish (3-5 minutes), monitoring port counters
Now do your maintenance
Verify connectivity is restored
Un-cull the BGP sessions - once you are ready
Watch traffic return

Design of typical BGP culling ACL

To implement this on an IXP you will need to devise an appropriate BGP culling ACL. It’s worth noting that this should:

Only affect BGP to/from your IXP addresses (i.e directly connected routers)
Work bidirectionally - BGP sessions may be established in either direction
Work with your hardware - can be applied either in or out on the physical port
Affect both IPv4 and IPv6

Most IXP switch platforms can be made to accept an appropriate ACL; I suggest this is made part of the requirements for any future platforms IXPs may purchase.

Results and operational experience

This was initially tested at LONAP during 2013, and first announced to the operator community as a lightning talk at RIPE67 in Athens in late 2013.

The technique can now be considered mature.

We have discovered it is also particularly useful in incidences where maintenances are complex, for instance a software upgrade and a fibre ODF repatch, where the end-user connection may suffer multiple link flaps. With this system you may now restore traffic once your work is complete and you have verified your infrastructure is working correctly.

RFC8327/BCP214

After some years’ operational experience, it was time to codify these techniques into an RFC to provide clarity to network operators on the process. With help from co-authors Job Snijders, Matt Griswold and Nick Hilliard, RFC 8327 was released in March 2018. It explores the various methods of traffic control during maintenance activity, including the process above