This is the next post on a series of posts on NSX-T Edge Node design topologies. In this blog post I will describe an Edge Node design topology hosting a Tier-0 Gateway with Dynamic Routing with BGP. This Edge Node topology supports Active/Active and Active/Standby Tier-0 design.
With the Edge Node Virtual Appliances it is important to know on which vSphere Hosts the Edge Nodes are going to run, how many physical NICs are available and if the Edge Node is running on top of a VSS/VDS or N-VDS and how teaming is configured. In this design topology the Edge Nodes are running on hosts with one VDS with two Physical NICs.
The following topology is showing two Edge Nodes running on a vSphere Cluster with hosts configured with one VDS with two uplinks.
On the Top of Rack (ToR) switches VLAN 100 is configured as Geneve transport network to carry overlay traffic. Each ToR has a dedicated transit VLAN configured for North-South traffic and BGP peering with the Tier-0 Gateway. VLAN 101 is configured on ToR-A and VLAN 102 is configured on ToR-B. It is a best practice to have a dedicated transit network per router for deterministic traffic and failover.
The Edge Node VM must have Management, Overlay, and Uplink Router (North/South) connectivity. The management IP can use the same Port Group as ESXi hosts or any general management Port Group, which is dedicated for Management connectivity. In this design topology there is a Port Group created for Overlay (Overlay-PG) and one Port Group for each Transit network for External traffic (External-A-PG for VLAN 101 and External-B-PG for VLAN 102).
Depending on if you want to tag VLANs on the VDS Port Group or in NSX you should either configure the VLAN ID on the Port Groups or configure these as Trunk allowing the required VLANs and configure the VLAN ID’s on the Segments where the router Uplinks are being configured on.
I recommend to configure specific Failover Order on the Port Groups configuring only the NIC connected to that specific ToR for achieving deterministic traffic with North-South traffic. In this topology on the VDS this would make sure that North-South traffic from fp-eth1 on the Edge Node, connected to the External-A-PG, will go through Uplink-1 (vmnic0) and North-South traffic from fp-eth2, connected to the External-B-PG, will go through Uplink-2 (vmnic1).
In NSX-T three Transport Zones with a N-VDS are required to achieve the required connectivity. On the Edge Node you configure the Overlay Transport Zone and N-VDS which you are also using across Hosts to make sure Edge Nodes can reach Hosts on their downlinks using the Overlay Network. The External-A-TZ with External-A-N-VDS and External-B-TZ with External-B-N-VDS VLAN Transport Zones are required to be able to map fp-eth1 with External-A-N-VDS and fp-eth2 with External-B-N-VDS.
Two segments are required for the Tier-0 Gateway Uplinks running on the Edge Nodes. A VLAN 101 Segment in External-A-TZ and VLAN 102 Segment in the External-B-TZ Transport Zone.
When configuring the Tier-0 Gateway you select it to be configured as Active/Active or Active/Standby depending on your use-cases and requirements. For example Active/Active is required for ECMP designs with high-throughput and fast failover use cases, where Active/Standby would be required for use cases where stateful services like NAT, Firewall or LB are required on the Tier-0 Gateway.
The diagram below shows a topology of a Active/Active Tier-0 Gateway on a pair of Edge Nodes. The Tier-0 Gateway will require to have two uplinks configured on each Edge Node. BGP peering will be established from each Tier-0 uplink towards the ToR and when ECMP is enabled will result in having four active North-South flows with the physical network.
The diagram below shows a topology of a Active/Standby Tier-0 Gateway on a pair of Edge Nodes. The Tier-0 Gateway will require to have two uplinks configured on each Edge Node. BGP peering will be established from each Tier-0 uplink towards the ToR. Note that in a Active/Standby configuration with BGP the Standby Edge Node will prepend it’s Autonomous System (AS) Number three times to make that path is less preferred and making sure that all active network traffic is sent towards the Active Edge.
In conclusion the Edge Node VM provides a lot of design flexibility for your NSX-T design and while designing your Software Defined Networking solution with NSX-T it is important to understand the underlying platform and network fabric connectivity options. In a future post I will share how to configure the topologies shared in this post.
Links to other NSX-T Edge Design Posts