Anycast transmission and how it’s leveraged by Global LBs

Photo by Taylor Vick on Unsplash

This blog is in continuation to this one which discussed geographic routing or geo-location based proximity routing using DNS mechanism. Here I’ll be discussing another way of achieving geographic routing which uses Anycast deployment. And this technology is leveraged by most of the Global LB service providers like GCP and AWS.

In order to explain this we’ll first talk about some basic concepts and terminologies like the network transmission types that exist, Autonomous systems and Border Gateway Protocol. And later, we’ll tie it all together to understand how geolocation routing is achieved.

Network transmission types

• Unicast
• Multicast
• Broadcast
• Anycast

By all means, feel free to skip over to the Anycast part, since others are not related to the topic at hand.

Unicast

This is widely used and understood mode of transmission wherein we have one-to-one communication between sender and receiver. (Think client connecting to a server on a given IP address and sending some packets over some protocol e.g. http, smtp, etc.)

Multicast

This is similar to broadcast wherein a source would be transmitting message to a destination IP address and clients/devices interested in receiving the messages will connect to the given Destination IP address. Any message sent by source will be relayed on to the connected devices.

Broadcast

Here a message sent by sender node can be heard by all the nodes connected on the same broadcast network. This is different from mulicast since there messages are received only by the devices interested in receiving it.

Anycast

Here single IP address is assigned to a collection of servers which may be deployed in different data-centres located in different regions. When a client tries to connect and send packets to that IP address, the network sees multiple routes leading to that IP address and selects one of them which is topographically closest, and routes the packets accordingly.

We’ll now dive deep into how this works and to understand how Anycast works, it’s imperative that we understand Autonomous systems and BGP first.

Autonomous systems

An autonomous system can be an ISP or a corporate network or any other large network that controls a collection for routing prefixes or a range of IP addresses. An AS also controls the routing policy for nodes within its network.

BGP (Border Gateway Protocol)

BGP is a protocol designed to exchange routing and reachability information among autonomous systems on the Internet. It makes routing decisions based on paths, network policies, or rule-sets. [Wikipedia]

Internet is a network of networks and when one node wants to send message to another node it would need to know what path to take for doing so. So in a nutshell BGP is internet’s navigation system. It is responsible to find the an efficient path for messages to travel from source to destination over the internet.

Each AS in the system maintains a table of known routes and broadcasts it to the neighbouring AS and the neighbouring AS can further propagate the information. In the below diagram, we can see how AS1 has maintained known routes to its on the system. For example, for a node in AS1 to reach to another node in AS3, next hop is through the router R2 which is via AS2. Now similarly, AS2 will further have routing information to be able to connect to the node within AS3.

Now that the basic concepts are out of the way, let’s see how this all ties together.

At Priceline, we use Google’s Cloud load balancer service which provides a global Anycast IP address to act as a frontend. GCLB instances deployed across the various GCP regions across the world host this IP address. Let’s take the example of a user trying to connect to the Priceline website from New York.

As soon as the user request is initiated, the AS within which the user resides will look for available routes to the GCLB instance i.e. 23.45.6.10. And as shown in the diagram above, there are multiple possible routes over which the request can reach the destination. It will pick the route which is topographically nearest and route the connection through it. The LB will then go on and further route the request to an appropriate backend service to serve the request.

Conclusion

Anycast deployment has several implicit benefits such as:

• Latency improvement: Requests are routed over the optimal path to achieve lowest possible latency
• Reliability: If a DC is out of service for maintenance, requests are automatically routed to other healthy service instance.
• Load Balancing
• Prevention from DoS attacks

And Global LB service providers rely heavily on Anycast transmission to achieve the global scale and the various benefits that come along with it.