The overlay and underlay networks are quite difficult to understand at first but it is pretty easy, in this blog lets go over the differences between underlay and overlay networks also do a lab to make more sense.
This is also the continuation of the nuage SD-WAN lab setup.
Initially, I thought I would just install all the Nuage SD-WAN components, bootstrap the NSG and done. Though installation would be complete and the lab would be up and running, it wouldn’t be fair to you if I don’t show the overlay and underlay network differences using real scenarios.
You can still feel free to read on even if you haven’t done the lab 🙂
Differences between the overlay and underlay network
First, let’s look at the differences between the underlay and overlay network then we can head over to the lab. If you have been following my lab up to this point you are pretty much worked on the underlay network.
So what is the Underlay network?
Underlay network is a physical network that would let you connect between the networking devices, such as routers, switches or even firewalls, etc. It also uses the traditional networking mechanism to route the traffic between the hosts.
For example, below are the two network Services gateway’s or ( Routers if you are looking from traditional network perspective ) and it’s connected via an internet link, internet-1. you can consider this as an underlay network. Each link has its own public IP network.
These NSG’s could be anywhere in the world.
What is the Overlay network?
An overlay network is a virtual network which is routed on top of underlay network infrastructure, routing decision would take place with the help of software.
In the example below you can see the two private IP address spaces, 10.1.1.0/24 and 10.2.2.0/24 routed via public underlay which we depicted just above, using the overlay network. In nuage SD-WAN, it uses VXLAN as the overlay network traffic.
With a single underlay network, you can have multiple overlay subnets as well, see below, However, underlay is unaware about the overlay network subnets.
To make it simpler let me put it this way,
Have you ever worked in virtualization?
In virtualization, the host where you have the VM’s resides is called Host machine and the VM’s are called Guest VM’s. Similarly, This overlay and underlay network works the same way, the physical networks are called the Underlay network and the network which runs on top of the physical networks are called overlay network.
It made more sense now, does it?
Is these overlay and underlay networks are interrelated?
The answer is YES!
If the Underlay network were to go down the overlay network would also go down but if the overlay network were to go down the Underlay network is not aware of any of these network outages.
So if the underlay is the core network how do we avoid network outages on the underlay?
To avoid the underlay network outage, you can have multiple underlay network links like below for redundancy.
You can have one more internet link, Internet-2 as the secondary link for both the NSG’s.
Now you can make both the internet links to be either active, active, which means it can load balance the traffic or you can do Active standby meaning during the internet-1 is down the internet 2 would take over.
Basically the overlay network is not aware of any of these changes that occurred on the underlay network and it keeps forwarding traffic as long as the Underlay network is available.
This concept is also useful during the troubleshooting of the nuage SD-WAN network. You can also narrow down the issue based on which layer (underlay or overlay) has the issue.
Overlay network creation in Nuage SD-WAN
Below is the actual underlay network, The Branch 1 NSG is connected to two WAN links both from internet 1 and 2 respectively, so is the NSG2 at the branch-2.
How does the NSG1 at the branch one talks to the NSG2 at Branch2 ?
It uses VXLAN tunneling mechanism between these two branches and passes the overlay traffic.
The LAN side of these branches which is the overlay is not aware of the underlay network.
The connection between the Branch-1 and Branch-2 LAN side would go via the VXLAN tunnel.
What if you add one more branch to the topology?
All the branch networks become a full mesh topology and start to send traffic with each other.
As you can see in the diagram above, we are going to use two overlay subnets in this lab, like below.
10.1.1.0/24 – Branch1
10.2.2.0/24 – Branch2
Follow the below steps to create the overlay network in Nuage SD-WAN.
- Goto the Organisation and click on Networks
- Click on the plus icon to create L3 Domain, I am naming it as L3 Domain.
- Drag and drop the zone templates to the domain.
- Select the domain you just created and click on the Instantiate domain from down bottom left. I just gave the name Network.
- Now you can start drag and drop the subnet to the branch networks, eventually it would look like below.
- You need to attach the NSG’s to the subnet by attaching Bridge Vport.
- You just attached the branch-1 LAN side to the overlay, you may do the same steps for the Branch-2 as well.
- We successfully added the NSG to the overlay network.
Attaching the end-user machine on the Overlay.
We have already created a VLAN 111 and 222 for Branch-1 and branch-2 LAN side respectively on the underlay gateway as well as on the KVM hosts where we are going to deploy the branch hosts.
I have also deployed Centos and Ubuntu 19.04 for Branch1 and Branch2 respectively, let’s connect both to our network.
- Going to use Centos as the branch-1 machine, Open the Centos VM and click on properties. On the network select Bridge interface ‘br111’ for the host and apply the changes.
- Do the same thing on the ubuntu host which represents Branch-2 as well, but use br222 for the second host instead of br111.
Now, will you be able to connect the hosts between?
Of course not as we have not configured the IP address on these hosts.
Let’s configure the IP for the hosts from the VSD using the DHCP.
Configuration of DHCP in Nuage VNS
- goto the enterprise, click on Network, select the subnet and click on the DHCP icon.
- Create the DHCP scope like below, I intentionally left the first ten IP addresses from the scope.
- Do the same thing for the second subnet as well.
- Now you connect the network in the hosts you would get IP address from those DHCP scopes.
As you can see below, I got an IP address 10.1.1.94 for the Centos from branch-1.
And got 10.2.2.235 for the Ubuntu from Branch-2.
Allow Overlay communication
In order for the branch to branch communication to happen, you need to allow the ACL policy in VSD.
- Login to the VSD and click on Policies, create an ingress security policy to allow-all like below.
- Do the same thing for the Egress security policies as well.
Overlay Communication verification
Lets start pinging the host’s machines.
As you can see I can ping the Branch-2 gateway IP 10.2.2.1 as well as the host IP which is 10.2.2.235
Lets verify the same from branch-2
Yes, I can ping the gateway 10.1.1.1 as well as the hosts 10.1.1.94 here as well.
Finally let’s also verify the traceroute of these packets.
When I ran my traceroute with the command mtr 10.1.1.94, you get the below output.
It is just overlay devices and there is no underlay networks in it.
Same on the other side as well with the command mtr 10.2.2.235