Introduction
In this blog post I would like to share how BGP is configured on VMware NSX to automatically update routing information with the physical network.
Design
Recently I was involved in a project where we used BGP to peer the NSX environment with the physical network. The design we did was challenging because of scale. The NSX environment we deployed meant for a multi-tenant Cloud platform scaled for several hundreds of tenants. We deployed a multi-site NSX environment. Each tenant has its own Universal Distributed Logical Router (UDLR) spanned across both datacenters, and an Edge Services Gateway (called the Tenant Logical Router or Tenant ESG) behind a Perimeter Logical Router (PLR or Perimeter ESG), which holds a maximum of 49 tenants or less depending the performance or bandwidth requirements.
In this post a reference design is used for a fictional tenant to proof the configuration and design.
Below this design is displayed. Note the the UDLR and Logical Switches are available on both sites. A tenant can deployed on both sites.
NSX multi-site design with BGP
Configuration
Logical Switches
First the necessary Logical Switches are created. The screen shot below shows the created Logical Switches on the primary site. Note the Universal Logical Switches which were created to make the network available on the secondary site.
Logical Switches
Universal Distributed Logical Router
An Universal Distributed Logical Router (UDLR) is deployed on the primary site and connected the associated Logical Switches. Local Egress is enabled.
UDLR Configuration on Primary Site
The interfaces for this UDLR are configured as shown below. Interface vNIC# 2 is the Uplink to the Tenant ESG. Note that interface #3 is the uplink for the secondary site to the Tenant ESG on that site.
ULDR Interface Configuration
Tenant Logical Router
Each Tenant is configured with a Edge Services Gateway (ESG) as the Tenant Logical Router which is the gateway to the tenant internal networks and services.
The ESG is deployed with the following configuration.
Tenant ESG Configuration
The interfaces are configured as shown below. The Tenant ESG inteface vNIC# 0 is configured as an uplink to the Perimeter ESG, which is configured later in this blog post. For scalability a /28 subnet is used to connect several tenant Tenant ESG’s to a Perimeter ESG. Interface vNIC# 2 is the link to the Tenant UDLR. Interface vNIC# 1 is reserved for future use (ECMP). Note that on interface vNIC#3 we’ve configured a Tenant BGP Test Public IP address (hidden in this screenshot) for a Transparant Mode Load Balancer.
Tenant ESG Interface Configuration
Perimeter Logical Router
The Perimeter Logical Router is the ESG which has an uplink to the physical network configured. This ESG also has BGP peering configured with the physical network. In this design the Perimeter ESG is used to scale up to 35 tenants per ESG. The Perimeter ESG is configured with HA and deployed as X-Large to have enough resources available to route and process all the traffic for the tenants.
Perimeter ESG Configuration
The interfaces are configured as shown below. Interface vNIC0 is configured as the uplink to the physical network. Because we are limited to 10 vNIC interfaces a Trunk is created on interface vNIC2 to scale up to 200 sub interfaces.
Perimeter ESG Interface Configuration
Each sub interface is configured as the transit link to each seperate Tenant ESG. In the example below interface vNIC10 is the transit link to the tenant ESG which was created.
NSX Edge Interface Configuration
Routing and BGP
Now BGP is configured on the UDLR, Tenant ESG and finally the Perimeter ESG.
UDLR
Edit the Global Routing Configuration on the UDLR on the Routing Page.
Routing Global Configuration
Edit the Dynamic Routing Configuration and select the Router ID.
Dynamic Routing Configuration
Continue on the BGP Configuration page.
BGP Configuration
Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.
Edit the BGP Configuration
Now the BGP neighbours are configured. In this configuration the Tenant ESG is the neighbour. The neighbour IP Address is 10.10.10.1 (Internal interface of the Tenant ESG). The forwarding address is the configured IP address of the UDLR and on the UDLR a Protocol Address is configured. The remote AS number is 65029 in this example.
BGP Neighbour Configuration on the UDLR
Continue on the Route Redistribution page and enable BGP.
Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.
Add BGP Route Redistribution Criteria
Tenant ESG
Continue with the Global Routing Configuration on the Tenant ESG.
Tenant ESG Routing Configuration
Edit the Dynamic Routing Configuration and select the Router ID.
Edit Dynamic Routing Configuration
Continue on the BGP Configuration page.
BGP Configuration
Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.
Edit the BGP Configuration
Now the BGP neighbours are configured. In this configuration the Perimeter ESG and the UDLR are the neighbouring routers. The neighbour IP Address is for the the Perimeter ESG is 172.16.20.1 (Internal interface of the Perimeter ESG) and the IP address of the neighbour UDLR is 10.10.10.3 (Protocol Address the UDLR).
Add UDLR Neighbour
UDLR Neighbour Configuration
Add Perimeter ESG Neighbour
Tenant ESG Neighbour Configuration
Continue on the Route Redistribution page and enable BGP.
Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.
Add BGP Route Redistribution Criteria
Perimeter ESG
Continue with the Global Routing Configuration on the Perimeter ESG.
Perimeter ESG Global Configuration
Edit the Dynamic Routing Configuration and select the Router ID.
Edit Dynamic Routing Configuration
Continue on the BGP Configuration page.
BGP Configuration
Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.
Edit the BGP Configuration
Next the BGP neighbours are configured. In this configuration the Physical Router and the Tenant ESG are the neighbouring routers. The neighbour IP Address is for the Physical Router is a Public IP address ending with .1 and the IP address of the neighbour Tenant ESG is 172.16.20.2 (Uplink interface on the Tenant ESG)
Add Physical Router Neighbour. Note that the Physical Router has a different AS configured.
Edit Physical BGP Neighbour
Add Perimeter ESG Neighbour
Edit Tenant ESG as a Neighbour
Continue on the Route Redistribution page and enable BGP.
Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.
Add BGP Route Redistribution Criteria
Routing and BGP configuration is complete.
Test the BGP Configuration
Open the console or a SSH session to the UDLR, Tenant ESG or Perimeter ESG to test the BGP configuration
Use the show ip route command to see if there are routes derived from BGP.
[text]show ip route[/text]
Show IP Route on Tenant ESG
Use the show ip bgp neighbours command to read detailed BGP information of the BGP neighbours
[text]show ip bgp neighbours[/text]
Test the routing with a ping to a VM in one of the Tenant networks behind the UDLR.
Ping Test from the Tenant ESG to a VM behind a UDLR
