Discover the difference between overlay and underlay network – Part-8

Saifudheen SidheeqLatest Posts, Nuage-SDN2 Comments

The overlay and underlay networks are quite difficult to understand at first but it is pretty easy, in this blog lets go over the differences between underlay and overlay networks also do a lab to make more sense.

This is also the continuation of the nuage SD-WAN lab setup.
Initially, I thought I would just install all the Nuage SD-WAN components, bootstrap the NSG and done. Though installation would be complete and the lab would be up and running, it wouldn’t be fair to you if I don’t show the overlay and underlay network differences using real scenarios.

You can still feel free to read on even if you haven’t done the lab 🙂

Differences between the overlay and underlay network

First, let’s look at the differences between the underlay and overlay network then we can head over to the lab. If you have been following my lab up to this point you are pretty much worked on the underlay network.

So what is the Underlay network?

Underlay network is a physical network that would let you connect between the networking devices, such as routers, switches or even firewalls, etc. It also uses the traditional networking mechanism to route the traffic between the hosts.

For example, below are the two network Services gateway’s or ( Routers if you are looking from traditional network perspective ) and it’s connected via an internet link, internet-1. you can consider this as an underlay network. Each link has its own public IP network.

internet as underlay network between the NSG's

These NSG’s could be anywhere in the world.

What is the Overlay network?

An overlay network is a virtual network which is routed on top of underlay network infrastructure, routing decision would take place with the help of software.

In the example below you can see the two private IP address spaces, 10.1.1.0/24 and 10.2.2.0/24 routed via public underlay which we depicted just above, using the overlay network. In nuage SD-WAN, it uses VXLAN as the overlay network traffic.

Overlay network on top of underlay
The Greenline represents overlay network

With a single underlay network, you can have multiple overlay subnets as well, see below, However, underlay is unaware about the overlay network subnets.

many subnets over single underlay
Single underlay, multiple overlays subnet

To make it simpler let me put it this way,

Have you ever worked in virtualization?
In virtualization, the host where you have the VM’s resides is called Host machine and the VM’s are called Guest VM’s. Similarly, This overlay and underlay network works the same way, the physical networks are called the Underlay network and the network which runs on top of the physical networks are called overlay network.

It made more sense now, does it?

Is these overlay and underlay networks are interrelated?

The answer is YES!

If the Underlay network were to go down the overlay network would also go down but if the overlay network were to go down the Underlay network is not aware of any of these network outages.

So if the underlay is the core network how do we avoid network outages on the underlay?
To avoid the underlay network outage, you can have multiple underlay network links like below for redundancy.

You can have one more internet link, Internet-2 as the secondary link for both the NSG’s.

dual underlay for redundancy

Now you can make both the internet links to be either active, active, which means it can load balance the traffic or you can do Active standby meaning during the internet-1 is down the internet 2 would take over.

Basically the overlay network is not aware of any of these changes that occurred on the underlay network and it keeps forwarding traffic as long as the Underlay network is available.

This concept is also useful during the troubleshooting of the nuage SD-WAN network. You can also narrow down the issue based on which layer (underlay or overlay) has the issue.

Overlay network creation in Nuage SD-WAN

Below is the actual underlay network, The Branch 1 NSG is connected to two WAN links both from internet 1 and 2 respectively, so is the NSG2 at the branch-2.

differences between the underlay and overlay

How does the NSG1 at the branch one talks to the NSG2 at Branch2 ?

It uses VXLAN tunneling mechanism between these two branches and passes the overlay traffic.

overlay network using vxlan tunnel

The LAN side of these branches which is the overlay is not aware of the underlay network.

The connection between the Branch-1 and Branch-2 LAN side would go via the VXLAN tunnel.

What if you add one more branch to the topology?
All the branch networks become a full mesh topology and start to send traffic with each other.

As you can see in the diagram above, we are going to use two overlay subnets in this lab, like below.

10.1.1.0/24 – Branch1
10.2.2.0/24 – Branch2

Follow the below steps to create the overlay network in Nuage SD-WAN.

  • Goto the Organisation and click on Networks
  • Click on the plus icon to create L3 Domain, I am naming it as L3 Domain.
creation of overlay network in nuage
  • Drag and drop the zone templates to the domain.
Creation of overlay network in nuage
  • Select the domain you just created and click on the Instantiate domain from down bottom left. I just gave the name Network.
Creation of overlay network
  • Now you can start drag and drop the subnet to the branch networks, eventually it would look like below.
adding subnets to overlay networks.
  • You need to attach the NSG’s to the subnet by attaching Bridge Vport.
attaching the NSG to the overlay network.
  • You just attached the branch-1 LAN side to the overlay, you may do the same steps for the Branch-2 as well.
  • We successfully added the NSG to the overlay network.
branch creation in Nuage SD-WAN

Attaching the end-user machine on the Overlay.

We have already created a VLAN 111 and 222 for Branch-1 and branch-2 LAN side respectively on the underlay gateway as well as on the KVM hosts where we are going to deploy the branch hosts.

I have also deployed Centos and Ubuntu 19.04 for Branch1 and Branch2 respectively, let’s connect both to our network.

  • Going to use Centos as the branch-1 machine, Open the Centos VM and click on properties. On the network select Bridge interface ‘br111’ for the host and apply the changes.
attaching the end hosts in overlay network
  • Do the same thing on the ubuntu host which represents Branch-2 as well, but use br222 for the second host instead of br111.

Now, will you be able to connect the hosts between?

Of course not as we have not configured the IP address on these hosts.
Let’s configure the IP for the hosts from the VSD using the DHCP.

Configuration of DHCP in Nuage VNS

  • goto the enterprise, click on Network, select the subnet and click on the DHCP icon.
  • Create the DHCP scope like below, I intentionally left the first ten IP addresses from the scope.
configuring DHCP on nuage overlay network
  • Do the same thing for the second subnet as well.
  • Now you connect the network in the hosts you would get IP address from those DHCP scopes.

As you can see below, I got an IP address 10.1.1.94 for the Centos from branch-1.

overlay IP on the end host on branch1

And got 10.2.2.235 for the Ubuntu from Branch-2.

overlay IP on the end host on branch2

Allow Overlay communication

In order for the branch to branch communication to happen, you need to allow the ACL policy in VSD.

  • Login to the VSD and click on Policies, create an ingress security policy to allow-all like below.
Ingress ACL policy on nuage SD-WAN
  • Do the same thing for the Egress security policies as well.
Egress Security policies on Nuage SD-WAN

Overlay Communication verification

Lets start pinging the host’s machines.

As you can see I can ping the Branch-2 gateway IP 10.2.2.1 as well as the host IP which is 10.2.2.235

Overlay communication verifcation

Lets verify the same from branch-2

Yes, I can ping the gateway 10.1.1.1 as well as the hosts 10.1.1.94 here as well.

Overlay communication verification

Finally let’s also verify the traceroute of these packets.

When I ran my traceroute with the command mtr 10.1.1.94, you get the below output.
It is just overlay devices and there is no underlay networks in it.

Running traceroute on the overlay network

Same on the other side as well with the command mtr 10.2.2.235

Running traceroute on the overlay network