Introduction
In this blog post I would like to share how BGP is configured on VMware NSX to automatically update routing information with the physical network.
Design
Recently I was involved in a project where we used BGP to peer the NSX environment with the physical network. The design we did was challenging because of scale. The NSX environment we deployed meant for a multi-tenant Cloud platform scaled for several hundreds of tenants. We deployed a multi-site NSX environment. Each tenant has its own Universal Distributed Logical Router (UDLR) spanned across both datacenters, and an Edge Services Gateway (called the Tenant Logical Router or Tenant ESG) behind a Perimeter Logical Router (PLR or Perimeter ESG), which holds a maximum of 49 tenants or less depending the performance or bandwidth requirements.
In this post a reference design is used for a fictional tenant to proof the configuration and design.
Below this design is displayed. Note the the UDLR and Logical Switches are available on both sites. A tenant can deployed on both sites.
Configuration
Logical Switches
First the necessary Logical Switches are created. The screen shot below shows the created Logical Switches on the primary site. Note the Universal Logical Switches which were created to make the network available on the secondary site.
Universal Distributed Logical Router
An Universal Distributed Logical Router (UDLR) is deployed on the primary site and connected the associated Logical Switches. Local Egress is enabled.
The interfaces for this UDLR are configured as shown below. Interface vNIC# 2 is the Uplink to the Tenant ESG. Note that interface #3 is the uplink for the secondary site to the Tenant ESG on that site.
Tenant Logical Router
Each Tenant is configured with a Edge Services Gateway (ESG) as the Tenant Logical Router which is the gateway to the tenant internal networks and services.
The ESG is deployed with the following configuration.
The interfaces are configured as shown below. The Tenant ESG inteface vNIC# 0 is configured as an uplink to the Perimeter ESG, which is configured later in this blog post. For scalability a /28 subnet is used to connect several tenant Tenant ESG’s to a Perimeter ESG. Interface vNIC# 2 is the link to the Tenant UDLR. Interface vNIC# 1 is reserved for future use (ECMP). Note that on interface vNIC#3 we’ve configured a Tenant BGP Test Public IP address (hidden in this screenshot) for a Transparant Mode Load Balancer.
Perimeter Logical Router
The Perimeter Logical Router is the ESG which has an uplink to the physical network configured. This ESG also has BGP peering configured with the physical network. In this design the Perimeter ESG is used to scale up to 35 tenants per ESG. The Perimeter ESG is configured with HA and deployed as X-Large to have enough resources available to route and process all the traffic for the tenants.
The interfaces are configured as shown below. Interface vNIC0 is configured as the uplink to the physical network. Because we are limited to 10 vNIC interfaces a Trunk is created on interface vNIC2 to scale up to 200 sub interfaces.
Each sub interface is configured as the transit link to each seperate Tenant ESG. In the example below interface vNIC10 is the transit link to the tenant ESG which was created.
Routing and BGP
Now BGP is configured on the UDLR, Tenant ESG and finally the Perimeter ESG.
UDLR
Edit the Global Routing Configuration on the UDLR on the Routing Page.
Edit the Dynamic Routing Configuration and select the Router ID.
Continue on the BGP Configuration page.
Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.
Now the BGP neighbours are configured. In this configuration the Tenant ESG is the neighbour. The neighbour IP Address is 10.10.10.1 (Internal interface of the Tenant ESG). The forwarding address is the configured IP address of the UDLR and on the UDLR a Protocol Address is configured. The remote AS number is 65029 in this example.
Continue on the Route Redistribution page and enable BGP.
Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.
Tenant ESG
Continue with the Global Routing Configuration on the Tenant ESG.
Edit the Dynamic Routing Configuration and select the Router ID.
Continue on the BGP Configuration page.
Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.
Now the BGP neighbours are configured. In this configuration the Perimeter ESG and the UDLR are the neighbouring routers. The neighbour IP Address is for the the Perimeter ESG is 172.16.20.1 (Internal interface of the Perimeter ESG) and the IP address of the neighbour UDLR is 10.10.10.3 (Protocol Address the UDLR).
Add UDLR Neighbour
Add Perimeter ESG Neighbour
Continue on the Route Redistribution page and enable BGP.
Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.
Perimeter ESG
Continue with the Global Routing Configuration on the Perimeter ESG.
Edit the Dynamic Routing Configuration and select the Router ID.
Continue on the BGP Configuration page.
Click on the Edit button to configure BGP. In this example the Local Autonomous System (AS) is 65029.
Next the BGP neighbours are configured. In this configuration the Physical Router and the Tenant ESG are the neighbouring routers. The neighbour IP Address is for the Physical Router is a Public IP address ending with .1 and the IP address of the neighbour Tenant ESG is 172.16.20.2 (Uplink interface on the Tenant ESG)
Add Physical Router Neighbour. Note that the Physical Router has a different AS configured.
Add Perimeter ESG Neighbour
Continue on the Route Redistribution page and enable BGP.
Add a Route Redistribution criteria in the Route Redistribution table. Select BGP as the learner protocol and make sure it is connected.
Routing and BGP configuration is complete.
Test the BGP Configuration
Open the console or a SSH session to the UDLR, Tenant ESG or Perimeter ESG to test the BGP configuration
Use the show ip route command to see if there are routes derived from BGP.
[text]show ip route[/text]
Use the show ip bgp neighbours command to read detailed BGP information of the BGP neighbours
[text]show ip bgp neighbours[/text]
Test the routing with a ping to a VM in one of the Tenant networks behind the UDLR.