How to: VMware VCF NSX backup to external destination

By default, VCF will back up NSX configuration to itself. However, we should point it to an external location.

This is a File-Based Backup of SDDC Manager and NSX Manager. When you first log into VCF SDDC manager, you will see an Orange banner with information to reconfigure it to point to an external file based backup for SDDC / NSX manager. First we will need to create a new user in vCenter with a new Group for the job.

https://docs.vmware.com/en/VMware-Cloud-Foundation/4.0/com.vmware.vcf.admin.doc_40/GUID-28DF03BC-654B-4E41-A3C1-230B356E87C6.html

Log into VCenter and navigate to administration > Single Sign On > Users and Groups and create a new user and give that user a password on the vsphere.local domain. Not the localos domain.

Create a Group and put that user in that Group.

That user we create in VCenter will be used to perform the Authentication from SDDC manager.

Scroll down to see that field, and you will need to remember an Encryption Passphrase. I made it the same as my NSX-T enable passphrase, but you do you.

Once we click save it will either like it or tell us something went wrong and we need to correct it in order to be submitted and enabled. You should see it configuring the backup on successful form completion.

It starts configuring a backup configuration job we can expand in SDDC manager.

We can dig into the task by expanding tasks view and double click the backup task, we can explore the sub-tasks as they progress.

I like how SDDC manager we can make the tasks screen large and click on a tasks to view sub-tasks. After a minute or two, the Backup configuration tasks will complete.

If we check our external SFTP destination, we will see a backup file.

We reconfigured SDDC manager and NSX-T to an external location using SFTP.

https://docs.vmware.com/en/VMware-Validated-Design/6.0/sddc-deployment-of-the-management-domain-in-the-first-region/GUID-187F567B-13EB-493F-8B4F-520E1C4CAED4.html

Deploy onprem cloud with VMware VCF 3.9.1 on VxRail 4.7.410

Do the VxRail bring up till you see Hooray! This assumes you have plumbed your physical networks with VLANs and an upstream router. The VxRail validation check prior to deployment verifies ip’s subnets, gateway, vlan’s and reach-ability. You will have VLAN for management, vMotion, vSAN, and guest or prod / dev VM_network.

Once VxRail initial deployment is complete, rename any components to meet your standard. I renamed my cluster.

You will need to also externalize vCenter. Go to vCenter > DataCenter > cluster name > VxRail > System and put in your vCenter credentials to externalize.

Deploy cloud builder OVA, power it on. You’ll need to specify domain name, IP address FQDN etc. The version of cloudbuilder is 2.2.1.0 build 15345960.

Open a browser and go to the cloudbuilder IP address. Login as admin and the password you provided during the OVA deployment.

Read and agree to the EULA.

Select platform VMware Cloud Foundation on Dell EMC VxRail.

To learn more: https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/com.vmware.vcf.vxrail.admin.doc/GUID-A5029B3C-F47C-43D1-BA12-AC3ADAE2B765.html

Review all requirements, check the box and select Next.

Make sure you enabled SSH and the service runs on host startup.

Back in cloud builder, you clicked Next and we can now download the spreadsheet to import into cloudbuilder once we’ve filled it out.

The spreadsheet is quite detailed, but if you make any typos, the validation will make sure you fill it out correctly.

Don’t forget to configure BGP on the upstream router for both Edge Service Gateway link 1 and link 2. This is a total of 4 BGP neighbor configurations that need to be configured upstream. Good NSX diagram is located at: https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/com.vmware.vcf.ovdeploy.doc_39/GUID-2F9F8BA4-D0EC-42B3-8AE9-D7B82C9EA54D.html

Upload the spreadsheet into cloudbuilder. Other format besides .xlsx is JSON.

With any luck your cloud builder will validate successfully. Verify FQDN, reverse DNS lookups, IP address, subnets again and double check it. Here’s my DNS so far.

Luckily, I’ve configured a few and I actually got it to validate on the attempt in a brand new setup. Click Next after successfuly validation. If there are any issues during the validation, it must be addressed. Typical issues are with FQDN’s, NTP, subnets, vlan’s and CIDR’s not matching up.

VCF Validation results download from image above is posted below.

VCF Validation results.

Click Deploy SDDC.

Only thing left to do is monitor the deployment. Check on it periodically over next few hours. Speed of deployment varies, but about two hours. It’s possible you may run into BGP neighbor configuration issue as all 4 paths must be in a BGP ESTABLISHED state. I’ve had to click retry before. If it still doesn’t work with one retry attempt, might log into NSX and or the upstream routers and verify ping works and show bgp neighbor has all routes established state. The upstream router must also advertise routes. If you make it past BGP then you are on the home stretch. This is typically where I have trouble and I will either log into a ESG console or the upstream routers and verify the ASN number, IP address of neighbor and password. The password field is optional in this release. However, it’s best to get familiar with configuring BGP with password enabled. This means, setting a BGP password on the upstream router.

Click Finish once VMare VCF bring up completes.

Click on Launch SDDC Manager to launch VCF.

There a good message and link to install Skyline for proactive support. Skyline: https://docs.vmware.com/en/VMware-Skyline/index.html

Log into VCF and enable CEIP. Navigate to Repository Settings and input your VMware and DellEMC support credentials to enable downloading bundles for VMware VCF / DellEMC VxRail.

After you input your VMware support and DellEMC support credentails, you will see bundles available for download under Repository > Bundles.

Scroll down and we see VxRail Software Update 4.7.511 as well as VCF bundles 3.10 and higher updates. There are plenty of hyperlinks linking to release notes. Some patches are for installs, some are for updates. Select Download Now or Schedule the download for later. Once we have bundles downloaded, we can then begin Life Cycle Management updates for VCF on VxRail thru VCF/SDDC manager. Now is a good time to also see about the messages at the top of the screen. The orange part says we should backup NSX Manager to an external SFTP server. Doing the NSX Manager backup will be part of my next blog.

Bringup Results information from Cloud Builder.

Bringup Results information from Cloud Builder.

PowerSwitch 5248F-ON Let’s make some VLAN’s

Before we begin, it’s always a great idea to have a physical drawing or make a physical drawing based on information you eyeball or able to gather using show lldp neighbor commands. Below is a drawing I created, that I will be using for a guide. In this demonstration, we will be focused on the two switches connected to the VxRail. The port-channel information, shown below as “Po” is shorthand for port channel and can be any number between 1 and 128. The upstream port-channel doesn’t necessarily have to identified as the same port channel number. Port channel numbers are locally significant.

Login to the switch and run a few commands to get a general sense of the switch config.

Looking at the running config, i’m going to use some existing vlan’s as a template to make my new vlan’s.

S5248F-FR8U13-site2# show running-configuration

Taking some and copy into notepad to create your new VLAN’s. We will edit in notepad to create new VLAN’s to paste into the switch config. Below are the new VLAN’s I want to add (2641-2648). I give them some descriptions to better document how the VLAN will be consumed. Next week, I might forget and this will help me and other’s understand the topology.

From switch CLI enter into configuration terminal mode. Then copy from notepad into the switch console.

S5248F-FR8U13-site2# configure terminal

We set the MTU size, and did a no shut to make the VLAN active. We still need to configure the VLANs on the upstream port channel and the server facing ports.

To make things easy, I’m going to grab one of the ports that connected and configured already and use that to help me configure the interfaces.

Copy the output of the interface shown above and put in a text editor and add in our vlan’s 2641-2648.

Since we just want to update the vlan’s instead of copying the entire interface configuration, i’m just going to grap the switch port trunk allow vlan and paste that updated bit into my switch.

Run the show interface command again on the switch to verify which ports we want to update. We will run the updated VLAN command against ports eth 1/1/5 – 1/1/12 and again for 1/1/16 – 1/1/24 for consistency. One can argue it’s not needed on some of these ports, but I like consistency.

When in the switch cli you can often get more information about command structure by doing ‘tab’ ‘tab’ like below. I did switchport trunk allowed vlan ‘tab’ ‘tab’ or question mark ‘?’ to get more information on correct syntax. The ‘tab’ key will autocomplete any partial command you’ve typed and the question mark will give more details on the next option.

Let’s look at the interface status command again to verify VLANs are there.

Now we need to tag the upstream port channel. Find the port-channel and add those new VLANs to the port-channel. Below we see how this is done.

You might notice there’s two port channel’s port channel 44 and port channel 1000. I have previously configured VLT on these switches. If VLT is configured, the new VLANs will automatically be added to the VLT port channel, once both switches in the VLT pair have the vlan’s configured.

Once you are satisfied with your updates save the running config to the startup config

S5248F-FR8U13-site2# copy running-configuration startup-configuration

Now we need to exit out of this switch and login to the other switch in the VLT pair. If you don’t have a VLT pair, you should at least have a secondary switch so there’s no single point of failure. Typically there is always two switches for redundancy. Often we will say Top of Rack 1 and Top of Rack 2 and shorten that down to ToR-1 / ToR-2 when speaking. I like to give the switch a meaningful name that shows the Rack number and the U number in the rack. In this example, it’s Front Row rack 8 in U 12 / U 13.

We should still have the switch configuration in our notepad to easily update the secondary switch, ToR2, hostname: S5248-FR8U12-site2.

Above, I ran a few commands like show version, show vlan, show interface status to get a feel for the configuration. Let’s run configuration terminal to get into configuration mode and run the commands from notepad.

Update the server facing ports. If I just add the new VLANs it will not remove the existing VLANs. Notice this method is slightly different than I describe previously. Instead of listing out all existing VLANs, I just type the ones that I want to add.

Update the upstream port channel as well. Add VLAN 2641-2648 to port-channel 44. Your VLANs and port-channels may differ, as each environment is unique, but you get the idea. One call out, notice on the port-channel running configuration, there’s a parameter saying “vlt-port-channel 44”. Since the switches are in a VLT pair, this allows me to create a unified / single port-channel that spans both switches in the VLT pair. Note: There can only two switches in a VLT pair. Other use cases for VLT might be a server facing port channel. In my scenario, I have a VLT pair going upstream to a secondary pair of switches in a VLT pair. This allows me to have path redundancy and no single point of failure between two sets of switches.

Don’t forget to save the running configuration to the startup configuration.

S5248F-FR8U12-site2# copy running-configuration startup-configuration

We have successfully added the new vlan’s to the PowerSwitch S5248F-ON in a VLT pair.

To touch on the comment about the VLAN being automatically added to the VLTi port-channel, if we run show vlan again on either switch, after doing both sides we will see that the newly created vlan’s are added to the VLT port channel as below.

As a closing statement, the only thing left to do is make sure the VLANs exist on the upstream switches all the way to the core gateway / router. In some cases, the gateway / router might be on the local ToR’s or upsteam somewhere.

Thanks,

Update VMware VCF 4.0.0.1 to 4.0.1 on VxRail.

Log into VCF and put ensure there are valid credentials to download bundles by selecting Repository Settings and verify My VMware Account Authentication.

Select Repository > Bundle Management and download the bundle.

After bundle download completes, navigate to Inventory > Workload Domains and select the workload domain to be updated. Notice far right, there is a Update Available column and it says Ready. We will select the management domain. The management domain must be updated prior to any other workload domain.

Select Updates / Patches to get more details.

From the Updates / Patches menu, we can scross down and view Current Versions. We can also perform a Precheck prior to beginning the update. It’s always a great idea to peform the precheck.

After clicking on Precheck, we can click on View Status to get more information.

While precheck is running we can review details and expand sections of the precheck to get more information about a particular step in the process.

Here I have expanded SDDC Manager > Domain Manager to learn more about this process.

Similarly, we can expand LCM for Life Cycle Managment information.

VCenter check as well. Sometimes, you might see that a password has expired or NTP is not working. If there are any issues during precheck, they must be addressed prior to performing the update. Luckily, the precheck does a great job of letting know about issues prior to preforming the update procedure. Perhaps someone disabled DRS? Did someone create a host affinity rule pinning a guest VM to a host, thus preventing a host to enter maintenance mode? If a host is unable to enter maintenance mode, Precheck will tell you why the operation has failed.

Here we see the Precheck operation was a success. click on exit to exit the precheck screen.

Since the update precheck passed, we can schedule the update for some time in the future or click Update Now. Note, we can run precheck as many times as we like. It doesn’t necessarily mean we will do the update.

If we click Schedule Update will be be shown the following menu about scheduling.

I’m going to cancel out of the Schedule Update menu.

Once we are satisfied click Update Now.

After a moment or two we will see the screen change and begin the update process. “VCF Upgrade will start in less than one minute”

Next a new screen will be presented as it walks down the list of things to update.

Here I am expanding the Domain Manager section to view just a bit more information.

Now would be a great time to read the release notes if you have not already. https://docs.vmware.com/en/VMware-Cloud-Foundation/4.0.1/rn/VMware-Cloud-Foundation-401-Release-Notes.html

If you click on the link to “View Update Activity” you will see more information about the processes.

Update times can vary based on the update and cluster size and other factors, but eventually it will complete. Below, we can see this update took almost 29 minutes. Click Finish.

If you want to view more information about the update, at some point later, we can always go to Workload Domain > select the domain and from there select > Update History to get more information about past updates.

If I select View Update Status, we can see the actions performed and results. Click Exit Status to close the status menu.

Thanks,

Octant install for visibility into kubernetes on VCF VxRail

I will be doing this from the context of a Windows jump VM. It’s running Windows Server 2019.


Open PowerShell and Run as Administrator. Install Chocolately so we can install Octant. Instructions here:

https://chocolatey.org/docs/installation


Chocolatey (Choco.exe) command is now installed. Let’s close this windos and open a new instance of PowerShell as Administrator.

New instance of PowerShell shows choco is ready.


Let’s install octant using choco.


Normally I run kubectl commands from a linux sandbox vm, but i’ll need to install kubectl on this Windows system in order for this to work. So, let’s visit the vCenter select Menu > Workload Management to get the IP address of kubernetes, 172.28.76.1.


Follow instructions to download kubectl from that IP. Open a web browser and go to the kubernetes IP address using https, not http. Follow instructions there to install kubectl.


Download the zip file for Windows and put the binary files.


Open PowerShell again.. this time as your regular user account, not administrator. Run the kubectl command to authenticate to kubernetes.


Let’s run octant now.


The octant UI is now available. Let’s open a browser and go there.

Now that we have the UI up, let’s switch context to demo1 by changing namespace at top right of screen. I am changing to demo1 namespace. Here we can see an app running called yelb.


Let’s make a VLAN backed NSX-T logical segment.

The setup is deployed on VMware VCF 4.0 on DellEMC VxRail 7.0. As part of the bring up, you have enabled (Application Virtual Network) AVN networks. This is a fancy way of saying the Tier 0 and Tier 1 router have been deployed and configured to allow network segments to span to different regions. In this type deployment, we can put our vRealize suite on a AVN network. In times past, this would have been a VLAN backed network, but will not be backed by a NSX-T GENEVE network. If that’s unfamiliar, it’s a replacement to VXLAN network definition. Going back further, there’s VLAN’s isolate traffic into a “virtual” network, which provides security and limits broadcast, multi-cast, and uni-cast to an isolated space. This is Layer 2 space in terms of the OSI model defined by IEEE 802.3 standard. Layer 2 is the Data Link Layer logic that happens between connecting a physical cable to port and sending Layer 3 IP packets. It must first get a link up established and then the system is able to send / broadcast frames which are assembled on the far side to form an IP packet.

Some services / protocols / bits at this level include MAC address (Medium Access Control) which is a physical address, much like an IP address (layer 3), with a distinction, that a MAC address is defined by firmware or a virtual mac. Often in logs you will see something like PHY signifying a physical interface associated with a MAC address. Windows calls is the Physical Address. Linux calls the MAC a link or ether address. Protocols include LLDP (Logical Link Discovery Protocol), and CDP (Cisco Discovery Protocol). Other things happening at Layer 2 is flow control, acknowledgements, and back off timers to prevent and detect collisions in terms of “when to speak”.. so everyone is not talking at same time on that cable/link. Switches operate at Layer 2 and allow multiple conversations to flow on each unique wire. Switches offer frame buffering to ease congestion and enables ability for multiple computers to speak simultaneously, in that they are connected to the switch on a different physical wire (CDMA/CD). Other benefits as well..

VMware with assistance from other network vendors create a application virtual network similar to a traditional frame, but encapsulated with variable length messaging and different messaging fields to enable frames to flow across a Layer 3 IP address. Well, that’s interesting.. Sending a Layer 2 packet across a Layer 3 link. But how? The Layer 2 over Layer 3 frames are virtualized using Geneve standard network overlay encapsulation.

Reference: https://datatracker.ietf.org/doc/draft-ietf-nvo3-geneve/

NSX-T 3.0 supports virtual networks outside the constructs of physical vlan’s, but also allow ability of a virtual machine to be attached to a VLAN backed network segment, or LAN even though the VM is a, well, virtual machine. More recently, one can install the NSX-T driver on a physical Windows or Linux system to have it participate in Geneve virtual network conversations, but let’s focus on getting a VM to talk to a physical device.

Log into NSX-T and review the host transport node configuration. The cluster will show two transport zones, one for overlay one for vlan. We will remember the vlan backed profile to create our vlan backed NSX logical segment.

In NSX-T, select Networking > Segments > Add Segment. Give it a name and select the transport zone from the dropdown menu. This will be what we saw for the vlan backed host transport nodes above.


Back in VCenter, we can see a new portgroup with VLAN ID 224 defined.


Create a VM or move a VM to the new NSX-T VNI segment. Notice in Vsphere 7 with VDS 7, the ‘N’ in the portgroup naming convention, signifying a NSX segment.

Below, I’ve created a CentOS 7 VM and attached it to the NSX logical segment called VL224.


I have a DHCP server on VLAN 224, so it should pick up an IP automatically, assuming the VM is set to DHCP. Below we can see we did get an IP on vlan 224 and we can also see the MAC address of the VM. The MAC address is important in that will will let us know that yes we are talking to that VM when we review the Address Resolution Table (ARP). This tells which MAC has what IP address.


If we go to a device outside of VCenter, on VLAN 224, let’s see if we can ping this VM. I went to an old physical server connected to a physical port on VLAN 224. We verify the IP address, ping the VM, and check the ARP table. The MAC address matches what we expect. We are talking to that VM and that VM is on VLAN 224.


Now we have a VLAN backed NSX-T Logical Segment created. We verified it’s on the network, ping worked, verified the MAC addess. If ping doesn’t work the first time, don’t give up, verify there’s no firewalls preventing ping as well. By default there is not a rule in NSX preventing it. If needed, you may need to go to the physical switches and verify the physical switch see’s the expected MAC’s, and all switches between the VM and the physical machine have the VLAN configured.

It’s always a great idea to have a physical diagram (Layer 1), Data Link Layer (layer 2) consisting of MAC addresses and the Logical Link control layer. Verify VLANs, MAC addresses, and any interconnecting port channels across the physical fabric. Lastly, once physical layer and logical layer is defined, then we can verify MAC’s are seen at far side and near side of conversation and then look at Layer 3 IP communication.

This will require getting information from the physical switches to troubleshoot.

One can check that they see both MAC’s on same switch and from there check interconnected switches. Below, we see the MAC address of the VM and the other physical server. The VM is seen on eth 1/1/5 and the physical server is across a port channel somewhere else. If you don’t see both MAC’s, then you’ll look at your interconnects for VLAN tagging. If there are still issues, verify spanning tree is configured and both switches have same root bridge.

One can also verify that that vlan is allowed on the physical port and the upstream port-channel.


Thanks,

Kubernetes: VCF 4.0 on VxRail 7. SSH into a pod

The last blog post, I showed how to provision a pod containing SSH service. I want to do this so that I can then get basic dial-tone access into a pod so I can test and view networking aspects of the pod. This will help me get a better understanding of what’s going on with the container as it relates to networking.

To review, we created a Docker file and uploaded to Harbor or Docker.com and then we created a yaml file pointing to the image. We then apply the kubernetes yaml file to deploy the application.

From VMware VCenter, under Hosts and Clusters, we can see the sample pods under namespaces. I created a namespace for demo2 and a namespace for demo3.


We then go to our linux client and login to the kubernetes environment. You can install the client on a Windows or Mac system just as well. I created a alias link from kubectl to ‘k’ for brevity.


We can see the external IP is 172.28.84.2. The yaml file apply to create the cluster is published to github. One can view and download the yaml file here. https://github.com/batx123/k8s-sshd

The yaml file will pull from docker.com / docker.io so if your cluster has internet access, then it should work fine in your setup.

Now that we have the External IP info, let’s SSH in as root using VMware1! for the password.

Ok. That works. Let’s check demo3 or recreate it..

But first we need to switch context to demo3.


OK. let’s change it up a bit and create the cluster by running the yaml file directly off of github.

$ k apply -f https://raw.githubusercontent.com/batx123/k8s-sshd/master/egsshd.yaml


As we can see, we created a new cluster on demo3 and we have a external IP of 172.28.84.5. Let’s log into that pod.


So far nothing new, we know we can log into both pods in their respective namespace. Now, let’s see if I can get more details from the pod in demo3. But first, I’ll need to install some tools as they are not installed by default. We want to install ping and view ip address.

~# apt install iproute2 iputils-ping -y

Once we install those packages we can run ip address. Let’s install the packages and try to ping the other pod in namespace demo2 (172.28.84.2).

We can see the ip address, which is the internal networking of 10.22.1.18, and we can ping external ip of the other pod in demo2. Can we ssh from one pod to the other? Yes we can. While i’m in this pod, I install the ping and ip addr tool here as well.


172.28.84.2 is demo2


172.28.84.5 is demo3

Let’s try using the internal IP’s assigned by kubernetes instead of the loadbalancer / front end IP.

demo2 egssh pod (egsshd-7c78696976-sq46t) internal IP is 10.22.1.2/28

demo3 egssh pod (egsshd-7c78696976-twn2f) internal IP is 10.22.1.18/28


We cannot ping from one pod, internal IP, to the other as they are in different namespaces (demo2 and demo3).

The cluster IP’s are not pingable as well. So far we’ve only been able to ping the external IP. Sounds reasonable so far.


What if we put two of these pods in the same namespace?

in order to do this, I edit my yaml file and change the name of the service from egsshd to egssh to make it unique. After I edit the yaml file, I then apply the yaml file again, against the same cluster.


After I edit the yaml file I then apply it. For transparency, i’ll show the yaml file I edited. (I removed the ‘d’ from sshd app name so that the name is unique, not egsshd.)


After the edit, we then apply the yaml file. What’s interesting, is we now have two cluster IP’s. We will have to circle back around on that later. Ignoring the Cluster-IP for now.


After installing ping, I can now ping the other pod in this namespace.


Since I can ping.. let’s try SSH.


Here we showed how pod networking is isolated on a per namespace. If one namespace wants to talk to a different namespace, then it must use the front end / external IP, which just so happens to be a NSX-T LoadBalancer IP.

now to clean up. removing the ssh pods from demo2.


Check VMware VCenter and it looks good. No pods in demo2 now.

Let’s switch context to demo3 and delete that pod as well.


Once again we log into VCenter and demo3 has no pods.


Thanks,

Push image to Docker.com and then deploy via Kubernetes.

The last article we deployed a image to a local Harbor repository. While some might consider that more challenging, I’m going to do something similar here.

  1. We will push our image to Docker.com
  2. We will then create a namespace in kubernetes and pull the image from Docker.

Go to Docker.com and create your account.


Using the same SSH server image we created in the previous demostration, we will push that image out to Docker. The image name is: egsshd This is my attempt at building a simple service to get familiar with how to interact with Docker, and kubernetes.

Once you have a docker account set up, we are ready to push our image. First we will need to make a tag for the image using docker tag command followed by docker push command. Since my docker username is benatx123, I put that in my command, followed by the image name, followed by a tag.


Now in docker.com under repositories, we have a new public image. If you want to make your image private, you may.

I added a comment/Readme letting others know this is a simple ssh server service and the root password is VMware1!

Now that our image is ready we need to build a yaml file and point to the image. In the last example, I just posted a screenshot of the yaml file. But, let’s make this a little easier. I will publish the kubernetes yaml file to github.com. I created a github account and then created a new repository called 8ks-sshd. You can find this repository here: https://github.com/batx123/k8s-sshd

Log into VMware Vsphere 7 with kubernetes and create a new namespace. Assign a user with edit rights to the namespace. Assign a storage policy to the namespace.


Log into kubernetes and apply the yaml file.


We created the deployment. Now.. does it work? Let’s see.


Boom! Mission accomplished. Now if you want to copy this.. just install git and do a git clone.

$git clone https://github.com/batx123/k8s-sshd.git


And if you are curious as to what happened in NSX-T 3.0, here’s a screenshot of the load balancer IP. The application was automatically configured by adding LoadBalancer to the yaml file.


Harbor registry on VCF 4 with Kubernetes

If you have previously reviewed my other blog post, you’ll notice that we have stood up VMware vCenter 7 with Kubernetes. VSphere 7 with Kubernetes provides first class native workloads in VMware much like VM’s. Now, what do we do? Well, we need to create or source a image from somewhere. Often folks will store images at various places like docker.io or something similar. What about just keeping my image local so one doesn’t have to storage the image on a public image site? Harbor registry service allows DevOps folks to push and pull images to a local registry and deploy pods using these images. Let’s see how this can be accomplished.

Log into vSphere and select Hosts and Clusters and navigate to the namespace part and enable Harbor.


You’ll need to specify a vSAN storage policy for Harbor.


After a minute or two you’ll see new namespace pop up on the left under Hosts and Clusters > Namespaces.


After all the pods and services start we should see a the Image Registry is healthy and running with a Link to Harbor UI.



Let’s log into Harbor using administrator@vsphere.local account

You can also click on the More info… link to be directed to a nice readme on github https://github.com/goharbor/harbor

Upon logging into Harbor, we see the namespace that I previously created called demo1. That’s interesting. We can click into demo1 and get more info about this namespace, but I’ll save that for next demo.


Going back into vSphere menu Workload Management, we see the new registry namespace there as well.


I’m not going to click around too much on this namespace, but we can see that some persistent volume claims were created.


Let’s go back to the Namespace view, and make a new namespace.


We make a new namespace, give permissions, and assign a storage policy to the new namespace, demo2. I’m going to skip over the permissions and storage selection as we already covered creating a namespace in previous blog.

Let’s flip back to Harbor and see what happened.

We now have a new namespace in Harbor called demo2.

Let’s publish something in Harbor. But first we need something something to publish. Let’s build a sample dial tone test using SSH to make sure we can get access to something after we publish and deploy into Kubernetes. You will need to stand up Docker somewhere to get a sample app going. Luckily Docker has a great example on this at: https://docs.docker.com/engine/examples/running_ssh_service/


Let’s build it following the instructions and you’ll see it go through the list of commands and output.

$ docker build -t egsshd .

For brevity, i’m jumping straight to the bottom of the docker build output.


We have a fresh image created with name egsshd

Let’s run the docker image to make sure we can get into it as expected.

Ok, it appears that we can SSH into this container. This will provide us with a basic dial tone service ensuring we do have access and things work as designed. Exit out of the image instance.

In order to log into Harbor, we first need to figure out how to either get a real SSL certificate.. but didn’t, so let’s accept a insecure or self signed certificate. https://docs.docker.com/registry/insecure/

I opted to accept the self-signed certificate of my Harbor appliance. To do that I plugged in my IP address of Harbor into the /etc/docker/daemon.json file. Follow that with restarting the Docker service.


Let’s login to Harbor. Login Succeeded. Hooray! no change to the images yet.. but let’s tag and push the image.


Next we will need to create a tag then push the image. Once we tag the image, then we push it to Harbor.


What happened in Harbor?

We login to Harbor using administrator@vsphere.local and we can see the repository count has incremented to reflect the push. If you are already in Harbor UI, click refresh


Click on demo2 and learn more about it. It’s taking up about 78MB of space.


Click on Logs and we can see the push operation.


Let’s login to kubernetes now. Get the control plane IP from Workload Management.


Let’s create a yaml file for the deployment.


We login to Kubernetes.

Switch context to demo2 namespace.


Create the cluster.


get pod status. Ready should be 1/1 and Status Running.


Now that we a running pod, let’s get the front end service IP


OK.. moment of truth… Let’s SSH into it.


Shazam!

Kubernetes and vMotion: what happens?

VMware 7.0 with kubernetes has been deployed.

Someone asked me what happens during a ESX vMotion? Does the pod migrate? not quite. While it will spin up a new pod on a different node, that’s not a vMotion. If you want that data to be persistent, make sure your application has a persistent volume claim.

Create a namespace

Give it a name


ReadMe


Add permissions, with Edit Role. I previously created a user account name demouser.



Add storage policy, typically vSAN Default Storage Policy. Other policy profiles can be defined as well thru vCenter VM Storage Policies menu.




From here, we can set any Capacity and Usage limits.


Let’s get logged kubernetes by following the “Link to CLI Tools” by either selecting the “Copy link” or “Open” to watch the installer for your client system to which you will connect from. I stood up a basic linux system for such purpose.

Open the link and review details.




Change dropdown from Windows to Linux or Mac OS

let’s get the plugin. There’s a nice how to there on that page when selecting CLI Plugin Linux.

I like to make a kubectl alias link to letter ‘k’. $ln -s kubectl k


Go to kubernetes.io and get a copy of the wordpress using persistent volume in mysql https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/

The long and short of it is make a directory and put the files in the directory.

After going thru the tutorial, I created my first github repo so I can redeploy a little easier. \n

located at: https://github.com/batx123/wordpress

$git clone https://github.com/batx123/wordpress.git

Connect to kubernetes

kubectl vsphere login –server 172.28.76.1 –vsphere-username administrator@vsphere.local –insecure-skip-tls-verify


change context to wordpress namespace

kick off the deployment using kubernetes command: $k apply -k ./


We ran get pods to ensure pods are in a READY STATE and also got the front-end IP

fill out the form to get wordpress initial config up.


Go ahead and do stuff to make it unique if you like, like a hello world page or something.


OK. Where is our data? Where is the the process running?

When we ran the kubernetes apply command, it created two persistent volume claims.

A different way of looking at same information is by selecting Hosts and Clusters > Monitor > Cloud Native Storage > Container Volumes.


Switch back to the linux cli and run the following command: \n

[ben@cent1 wordpress]$ k describe pod wordpress

The output is excellent source for reviewing pod details. We can see the pod is running on node: esx-08a.




Notice the image pull and successfully attached the persistent volume claim (pvc).

The Events section is telling us if things are working or if there was an issue in downloading image or attaching the newly created volume.


There are also Kubernetes and vCenter events in vCenter GUI by selecting the pod name under hosts and clusters > Monitor > Events



I am going to place host esx-08a in maintenance mode and rerun the describe command.

wordpress terminated the old pod on esx-08a, and it’s spinning up a new pod.


let’s run the kubectl describe pod command again, but this time, choose the updated pod name, the numbers and letters appended to the pod will be different.



Host esx-08a is still in maintenance mode. The following command shows more info on this node.


Let’s take host esx-08a out of mainenance mode


Let’s check on our website to see if it’s still configured and had my stuff. It does. My Hello World is still there. Edited page to add “In order for this to work, there must be a persistent volume claim.”