The largest size that a VMware VPN Network with a VMware SD-WAN Hub can be is limited by the Hub itself. For both scalability and risk reduction, it is better to handle the Edges with more than one Hub in big networks with thousands of remote sites. To do this, though, it wouldn’t make sense to make the customer run their own separate Hubs. Clustering together several Hubs makes it easy to use all of them at once and manages them as a single unit with built-in resiliency.
SD-WAN Edge Clustering solves the SD-WAN Hub scale problem by making it easy to increase the Hub’s tunnel capacity on the fly by connecting a group of Edges together logically. A group of SD-WAN Edges would provide resilience through an Active/Active High Availability (HA) structure, which is also provided by Edge Clustering. From the point of view of other Edges, a cluster works like an independent Hub.
You have the option of using either physical or virtual edges for the Hubs in a VMware Cluster. In the event that they are virtual, they might be located on a single hypervisor or on numerous hypervisors simultaneously.
Over time, each Edge in a cluster sends usage and load data to the SD-WAN Gateway. Based on how much the Edge’s CPU and memory are being used, as well as the number of tunnels attached to the Hub as a percentage of the Edge model’s tunnel capacity, the load value is found. Within the cluster, the Hubs don’t talk to each other directly or share details about their states. In data centers, Edge Clusters are usually set up as Hubs.
In theory, Edge Clustering could be used to make other vectors, like speed, bigger on the side. The current Edge Clustering implementation, on the other hand, was built and tested to only grow at tunnel capacity.
Working of Edge Clustering
The following are some key ideas that explain how SD-WAN Edge Clustering works:
- Hubs can use edge clustering in the following ways:
- To make it possible for a Hub to have a bigger tunnel capacity than what can be provided by a single Edge that is acting as a Hub.
- To distribute remote Spoke Edges among several Hubs and minimize the potential impact of an occurrence.
- The following is an example of how Cluster Score, a mathematical measure of system utilization, is calculated:
Central processing unit (CPU), memory (RAM), and tunnel capacity are the three metrics used to evaluate utilization.
- Each measure of usage is thought of as a percent, with 100% being the highest possible value.
- Based on the rated capacity for a specific hardware model or Virtual Edge configuration, tunnel capacity is determined.
- The aggregate of all three usage percentages produces an integer-based Cluster Score (1-100).
- CPU and memory usage show throughput and flow quantity on a Hub even though throughput isn’t directly looked at.
- For instance, on an Edge 2000:
20% of CPU time was used.
30% of memory is used.
600 connected tunnels, out of a total capacity of 6000, equals 10%.
Score for the cluster: (20 + 30 + 10)/3 = 20
If your Cluster Score is over 70, it means that your cluster is “over capacity.”
- A “logical ID” is a 128-bit UUID that makes a part in the VMware Network separate away.
One example is the use of logical identifiers (IDs) for both edges and clusters.The logical IDs are utilized for internal element identification and are guaranteed to be unique, even when the user is providing the names of the Edge and Cluster.
- Under default conditions, the load is spread uniformly across all Hubs. Because of this, it is essential that all of the Edges that are a part of a cluster have the same capacity and model.
- The WAN and local area network (LAN) interfaces will each be assigned their own unique IP address. All of the VMware SD-WAN Edges that are part of the hub cluster are required to operate a dynamic routing protocol, such as eBGP, with the Layer 3 devices that are located on the local area network (LAN) side. Additionally, each member of the cluster must have a distinct Autonomous System Number (ASN). On the local area network (LAN) side of the cluster, dynamic routing ensures that traffic from the DC to a specific Spoke site is routed through the member of the Edge Cluster that is most appropriate for that traffic.
Hub Edges in a cluster don’t use routing protocols or tunnels to connect or communicate with one another. They carry out data plane operations as autonomous Edges. When the Branch Edges are connected to various Hub Edges within the cluster, they rely on the LAN-side BGP peering to the core switch to manage Branch to Branch traffic.
What is the process by which the VMware SD-WAN Gateway track Edge Clusters?
When a Hub is added to a VMware SD-WAN Cluster, it will break down and rebuild tunnels to all of its given Gateways. It will also let each Gateway know that it has been assigned to a Cluster and give each Gateway a Cluster logical ID.
The Cluster’s SD-WAN Gateway tracks:
- The logical ID
- The name
- If Auto Rebalance is turned on
- A list of objects in the Hub that are part of the Cluster
The Gateway keeps track of each Hub object in the Cluster by:
- The logical ID
- The name
- Statistics that are sent from the Hub to each given Gateway on a regular basis and are updated every 30 seconds. Those statistics include:
- The Hub’s current CPU usage
- The Hub’s current memory utilization
- The Hub’s current tunnel count
- The Hub’s current BGP route count
Based on the above calculation, current Cluster Score.
Once it’s been seven seconds since the Gateway last got a packet from the Hub Edge, the Hub is taken off the list of Hub objects.
During the process of clustering, how are Edges assigned to a certain Hub?
An SD-WAN Orchestrator is responsible for supplying the Edge with the logical ID of the Hub to which it must be attached in a topology that is based on the classic Hub and Spoke architecture. The Edge will connect to the Hub by requesting connectivity information from its assigned Gateways about that Hub’s logical ID. This information includes IP addresses and ports, which the Edge will use to establish a connection to the Hub.
When connecting to a Cluster, this is the same action from the Edge’s point of view. It tells the Edge that the Cluster logical ID, not the individual Hub logical ID, is the logical ID of the Hub it should connect to. In the same way, the Edge sends a Hub connection request to the Gateways and waits for details about connectivity in response.
At this moment, there are two deviations from the basic behaviour of the Hub:
- First Divergence: The Gateway Has to Choose Which Hub to Assign.
- Second Divergence: The Edge’s various Gateways may assign it different tasks as a result of the first divergence.
Divergence Number One was first dealt with by assigning an Edge to the Hub in a Cluster that had the fewest tasks assigned to it using the Cluster Score. In theory, this makes sense, but in practice, it wasn’t the best solution because the Cluster Score is only updated every 30 seconds and a normal reassignment event can involve hundreds or even thousands of Edges. That is, if Hub 1 has a Cluster Score of 20 and Hub 2 has a Cluster Score of 21, all Edges would choose Hub 1 for 30 seconds. After that, Hub 1 might become too busy and need more reassignments.
The Gateway, on the other hand, will first make an effort at a mathematical distribution that is fair, disregarding the Cluster Score. In the event that there are sufficient Edges, the logical IDs of the Edges, which were produced by a safe random-number generator on the Orchestrator, will have an even distribution of values. In light of this, it is possible to compute a fair share distribution by making use of the logical designation.
- Assigned Hub index = Edge logical ID modulo the number of Hubs in the cluster
As an illustration:
- Four Edges with logical IDs that ending in 1, 2, 3, and 4
- Cluster with 2 Hubs
- 1%2 = 1, 2%2 = 0, 3%2 = 1, and 4%2 = 0. Note that “%” stands for the modulo process.
- Hub Index 0 is assigned to edges 2 and 4.
Hub Index 1 is assigned to edges 1 and 3.
This sort of assignment is more consistent than a round-robin type of assignment since it ensures that Edges will typically be allocated to the same Hub each time. This makes the process of assignment and troubleshooting more predictive.
Restarting a Hub (for reasons such as maintenance or failure, for example) will result in the Hub being detached from the Gateway and so removed from the Cluster. This indicates that Edges will always be distributed evenly following the restart of all Edges (as a result of the reasoning outlined above), but Edges will be distributed unevenly following any Hub event that causes it to lose connectivity.
If a Hub’s tunnel capacity is exceeded above the maximum allowed, what are the consequences?
The logic for assigning edges will try to spread them out fairly among all the available hubs. It will not be even on the Edges again after an event on the Hub, like a reset.
When Edges are first assigned, the Gateway usually tries to spread them out fairly among Hubs. An uneven spread is not a state that can’t happen. If the assignments aren’t fair but no Hub is over 70% of its tunnel capacity, the assignment is still acceptable.
Clusters may reach a point when an individual Hub has exceeded 70 percent of its allowable tunnel capacity. This could occur as a result of an event that occurs on the Hub or as a result of the addition of new Edges to the network. Regardless of whether or not rebalancing is enabled on the Orchestrator, fair share redistribution will be carried out automatically in the event that this occurs when at least one other Hub is operating at a capacity that is lower than 70 percent of its total capacity. The majority of Edges will keep their current assignments as a result of the predictive mathematical assignment that makes use of logical IDs. On the other hand, Edges that have been assigned to other Hubs as a result of failovers or previous utilization rebalancing will be rebalanced in order to guarantee that the Cluster will automatically return to an even distribution.
What occurs if a Hub scores more than the highest permitted Cluster Score?
The Cluster Score is not continuously updated and cannot be automatically calculated by the Gateway following an Edge reassignment, unlike tunnel percentage, which is a direct indication of capacity and can be acted upon immediately. It is updated every 30 seconds. You can tell the Gateway to dynamically try to shift the edge load for each Hub as needed by setting an Auto Rebalance parameter in the Cluster configuration.
If the Auto Rebalance feature is turned off and a Hub has a Cluster Score that is greater than 70, but the tunnel capacity is less than 70, then no action is taken.
The Gateway will transfer one Edge per minute to the Hub with the lowest Cluster Score if Auto Rebalance is enabled and one or more Hubs score higher than 70. This process will continue until all Hubs score lower than 70 or until no more reassignments are possible.
By default, Auto Rebalance is turned off.
What occurs if two VMware SD-WAN Gateways assign different Hubs?
Because of the nature of a distributed control plane, the Cluster assignment is being determined independently by each Gateway. Most of the time, Gateways will determine each Edge’s assignment using the same mathematical procedure. This cannot be guaranteed in situations such as Cluster Score-based rebalancing.
If an Edge in a Cluster isn’t already linked to a Hub, it will accept the assignment from any Gateway that replies. This makes sure that Edges are never left without a job, even if some Gateways are down and others are up.
If an Edge is linked to a Hub in a Cluster and gets a message instructing it to pick a different Hub, it is handled in the order of “Gateway Preference.” For example, if the Super Gateway is online, the Edge will only accept changes made by the Super Gateway. Assignment requests from other Gateways that are in conflict will be ignored. For the same reason, if the Super Gateway isn’t online, the Edge will only take reassignments from the Alternate Super Gateway. When there are no Super Gateways and only Partner Gateways, the Gateway Preference is based on the order of the configured Partner Gateways for that Edge.
When using Partner Gateways, it is necessary to assign the same Gateways to both the Hubs in a Cluster and the Spoke Edges. If this is not done, there is a possibility that a Spoke Edge will be unable to receive Hub assignments. This is due to the fact that the Spoke Edge is connected to a Gateway that is not also connected to the Hubs in a Cluster.
After a VMware SD-WAN Gateway goes down, what does it do?
Edges may be moved when an SD-WAN Gateway goes down if the most preferred Gateway was the one that went down and the next most preferred Gateway gave a different task. Hub A was given to this Edge by the Super Gateway, and Hub B was given to the same Edge by the Alternate Super Gateway.
In the event that the Super Gateway goes down, the Edge will switch to Hub B because the Alternate Super Gateway is now the most recommended Gateway for connectivity facts.
Once the Super Gateway is back to normal, the Edge will ask this Gateway again for a Hub assignment. In the above situation, the Edge shouldn’t have to switch back to Hub A, so the Hub assignment request includes the currently assigned Hub (if there is one). The Gateway checks to see if the Edge is already given a Hub in the Cluster. If that Hub has a Cluster Score of less than 70, the Gateway changes the Edge’s local assignment to match the existing assignment without going through its own assignment logic. This makes sure that when the Super Gateway recovers, it will give the currently connected Hub and keep its assigned Edges from failing over for no reason.
If one of a cluster’s hubs loses its dynamic routes, what would happen?
As mentioned above, every 30 seconds, the Hubs report to the SD-WAN Gateways how many dynamic routes they have discovered through BGP. The SD-WAN Gateways will failover Spoke Edges to another Hub in the Cluster with an intact routing table if routes are lost for just one Hub in a Cluster, either because they are mistakenly retracted or because the BGP neighborship fails.
The route count is based on the exact instant the update is made to the SD-WAN Gateway because it is sent every 30 seconds. In the uncommon case that a LAN-side BGP neighbor completely fails, users can expect failover to take 30 to 60 seconds because the SD-WAN Gateway rebalancing logic runs every 60 seconds. Rebalancing is only allowed to occur up to once per 120 seconds in order to provide all Hubs an opportunity to update the Gateways again after such an occurrence. Accordingly, users should anticipate that failover will take 120 seconds in the event of a subsequent failure.
For LAN side failure identification, routes sent by BGP over IPsec/GRE are not taken into account. When the BGP over IPsec/GRE connection fails, the problem is not seen by the LAN side, so it does not cause the cluster to fail over.
How do we configure routing on a cluster hub?
Because the Gateway can tell the spokes to connect to any Hub in the Cluster, all of the Hubs should have the same routing configuration. The spokes should be able to reach the BGP prefix 192.168.2.1 behind the hubs, so all of the hubs in the cluster should advertise 192.168.2.1 with the same route attributes.
Within the cluster implementation, the utilization of BGP uplink community tags is recommended. When redistributing routes to BGP peers, it is necessary to configure the cluster nodes such that the uplink community tag is created.
What occurs when a cluster’s hub fails?
The SD-WAN Gateway will fail over Spoke Edges after waiting seven seconds for a tunnel to be declared dead. This means that in the event that an SD-WAN Hub or all of its connected WAN links fail, users should anticipate that failover will take 7–10 seconds (depending on RTT).
