The setup is deployed on VMware VCF 4.0 on DellEMC VxRail 7.0. As part of the bring up, you have enabled (Application Virtual Network) AVN networks. This is a fancy way of saying the Tier 0 and Tier 1 router have been deployed and configured to allow network segments to span to different regions. In this type deployment, we can put our vRealize suite on a AVN network. In times past, this would have been a VLAN backed network, but will not be backed by a NSX-T GENEVE network. If that’s unfamiliar, it’s a replacement to VXLAN network definition. Going back further, there’s VLAN’s isolate traffic into a “virtual” network, which provides security and limits broadcast, multi-cast, and uni-cast to an isolated space. This is Layer 2 space in terms of the OSI model defined by IEEE 802.3 standard. Layer 2 is the Data Link Layer logic that happens between connecting a physical cable to port and sending Layer 3 IP packets. It must first get a link up established and then the system is able to send / broadcast frames which are assembled on the far side to form an IP packet.
Some services / protocols / bits at this level include MAC address (Medium Access Control) which is a physical address, much like an IP address (layer 3), with a distinction, that a MAC address is defined by firmware or a virtual mac. Often in logs you will see something like PHY signifying a physical interface associated with a MAC address. Windows calls is the Physical Address. Linux calls the MAC a link or ether address. Protocols include LLDP (Logical Link Discovery Protocol), and CDP (Cisco Discovery Protocol). Other things happening at Layer 2 is flow control, acknowledgements, and back off timers to prevent and detect collisions in terms of “when to speak”.. so everyone is not talking at same time on that cable/link. Switches operate at Layer 2 and allow multiple conversations to flow on each unique wire. Switches offer frame buffering to ease congestion and enables ability for multiple computers to speak simultaneously, in that they are connected to the switch on a different physical wire (CDMA/CD). Other benefits as well..
VMware with assistance from other network vendors create a application virtual network similar to a traditional frame, but encapsulated with variable length messaging and different messaging fields to enable frames to flow across a Layer 3 IP address. Well, that’s interesting.. Sending a Layer 2 packet across a Layer 3 link. But how? The Layer 2 over Layer 3 frames are virtualized using Geneve standard network overlay encapsulation.
NSX-T 3.0 supports virtual networks outside the constructs of physical vlan’s, but also allow ability of a virtual machine to be attached to a VLAN backed network segment, or LAN even though the VM is a, well, virtual machine. More recently, one can install the NSX-T driver on a physical Windows or Linux system to have it participate in Geneve virtual network conversations, but let’s focus on getting a VM to talk to a physical device.
Log into NSX-T and review the host transport node configuration. The cluster will show two transport zones, one for overlay one for vlan. We will remember the vlan backed profile to create our vlan backed NSX logical segment.
In NSX-T, select Networking > Segments > Add Segment. Give it a name and select the transport zone from the dropdown menu. This will be what we saw for the vlan backed host transport nodes above.
Back in VCenter, we can see a new portgroup with VLAN ID 224 defined.
Create a VM or move a VM to the new NSX-T VNI segment. Notice in Vsphere 7 with VDS 7, the ‘N’ in the portgroup naming convention, signifying a NSX segment.
Below, I’ve created a CentOS 7 VM and attached it to the NSX logical segment called VL224.
I have a DHCP server on VLAN 224, so it should pick up an IP automatically, assuming the VM is set to DHCP. Below we can see we did get an IP on vlan 224 and we can also see the MAC address of the VM. The MAC address is important in that will will let us know that yes we are talking to that VM when we review the Address Resolution Table (ARP). This tells which MAC has what IP address.
If we go to a device outside of VCenter, on VLAN 224, let’s see if we can ping this VM. I went to an old physical server connected to a physical port on VLAN 224. We verify the IP address, ping the VM, and check the ARP table. The MAC address matches what we expect. We are talking to that VM and that VM is on VLAN 224.
Now we have a VLAN backed NSX-T Logical Segment created. We verified it’s on the network, ping worked, verified the MAC addess. If ping doesn’t work the first time, don’t give up, verify there’s no firewalls preventing ping as well. By default there is not a rule in NSX preventing it. If needed, you may need to go to the physical switches and verify the physical switch see’s the expected MAC’s, and all switches between the VM and the physical machine have the VLAN configured.
It’s always a great idea to have a physical diagram (Layer 1), Data Link Layer (layer 2) consisting of MAC addresses and the Logical Link control layer. Verify VLANs, MAC addresses, and any interconnecting port channels across the physical fabric. Lastly, once physical layer and logical layer is defined, then we can verify MAC’s are seen at far side and near side of conversation and then look at Layer 3 IP communication.
This will require getting information from the physical switches to troubleshoot.
One can check that they see both MAC’s on same switch and from there check interconnected switches. Below, we see the MAC address of the VM and the other physical server. The VM is seen on eth 1/1/5 and the physical server is across a port channel somewhere else. If you don’t see both MAC’s, then you’ll look at your interconnects for VLAN tagging. If there are still issues, verify spanning tree is configured and both switches have same root bridge.
One can also verify that that vlan is allowed on the physical port and the upstream port-channel.