Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Overall Steps performed are as:


The full installation of the overcloud and the undercloud here is done all-in-a-single KVM Host running libvirt on CentOS7.


1) Preparation of the KVM Host 


The KVM Host is going to be the physical server on which the undercloud VM and the overloud VMs will be there. The KVM server will run the base minimal install of the latest CentOS7 and willbe installed with the RPMs for libvirt 

This primarily involves the following steps 
please refer:

See Link: KVM host preparations for having all the undercloud and the overcloud machines run as a VirtualMachine on the same


   - Setting up system proxy and the /etc/yum.conf proxy for the KVM Host
   - Updating the CentOS 7 Operating System 
   - Setting the hostname
   - Setting the Static IP on the KVM Host
   - Setting the SELINUX to permissive/enforcing mode 
   - Installation of the common utilities

See Link: Enabling the processor Virtualization on the KVM Host

See Link: Enabling the nested Virtualization on the KVM Host

See Link: Creation of the Virtual Network on the KVM Host for the overcloud and the undercloud Virtual Machines

See Link: Configuration of the OS Install Virtual Media on the KVM host for Undercloud Virtual Machine OS installations




2) Creation and Installation of the undercloud 


See Link: Creation of the Virtual Machine for the Undercloud and OS installation on the same


   
3) Preparing the undercloud node for the undercloud install


The basic OS preparations to be done on the undercloud server  before the undercloud install are done here

See Link: Preparation of the undercloud Virtualmachine OS for undercloud installation

The steps covered are basically 

Setup static network configuration on the undercloud server (if not done during the OS install)

• Setup the hostname of the undercloud server (if not done during the OS install)
• Setting up /etc/hosts on the undercloud server
• Disable the NetworkManager and Firewalld services and enable the network services on the undercloud server
• Setting the SELinux to permissive mode on the undercloud
• Set the http_proxy and https_proxy for the undercloud server (as the undercloud server is behind a proxy)
• Update the OS and reboot the undercloud server 
• Create the stack user for the undercloud install
• Install the RDO Tripleo Repositories on the undercloud server
• Enable the Tripleo Repositories as per the release of the Openstack (Pike here)
• Install the python-tripleoclient

4) Installing the undercloud

Please note that if your undercloud is behind an HTTP and HTTPS_PROXY, ensure that the undercloud server IP is added to the no_proxy configuration in the environement profile of at least the 'stack' user. 

Else, as the undercloud final steps may fail as the openstack services when trying to reach each others' endpoints will be directed to the proxy. 

If the proxy denies such access, the openstack services will nor work properly on the undercloud.

In the .bash_profile of the stack user (better to add in /etc/profile) 

you must add a line like 

no_proxy="127.0.0.1,localhost,::1,$(hostname -i)

The installation of the undercloud is covered in at 

See Link: OpenStack Pike TripleO undercloud installation on the undercloud server

5) Overcloud installation preparation


See Link: Setup the DNS NameServer for the Overcloud provisioning Network on the Undercloud server 

See Link: Download or Build the overcloud images for the overcloud deployment

See Link: Upload the ovecloud images as the Glance images to the undercloud

See Link: Creation of the Virtual Machines on the KVM Host for the overcloud

See Link: VirtualBMC Setup on the KVM Host for the Virtual power Control of the overcloud guests

See Link: Creation of the JSON file to import the overcloud VirtualMachines as the baremetal node and Import for overcloud provision


See Link: Tune the undercloud flavors to match the overcloud node hardware configurations



See Link: Associate the baremetal servers to the correct profile and run introspection on the overcloud nodes finally set them to available mode

   

6) OverCloud installation:
   

See Link: Final preparations before running the overcloud deploy

See Link: Running the pre-validations and finally the overcloud deploy

7) Common tripleo deployment failures 

See Link: Common TripleO deployment failures

Setting the hostname and the static IPs on the KVM host

Setting the hostname for the KVM Hypervisor.

Ensure that the hostname has been set using

hostnamectl set-hostname lab.example.com

Confirm the same using hostnamectl command, Also put an Entry of the IP address and the hostname in the /etc/hosts of the KVM Host.


Setting the Static IP on the KVM Host

Ensure that the IP address has been set for the KVM Host 

Set a static IP in the /etc/sysconfig/network-scripts/ifcfg-<DEVICE>

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Setting up system proxy and the /etc/yum.conf proxy for the KVM Host (opestack Pike tripleo overcloud and undercloud)

Setting up system proxy and the /etc/yum.conf proxy for the KVM Host

As the KVM host is behind a proxy server, the needed proxy server was added to the KVM host /etc/profile and also to the /etc/yum.conf 


These lines has been added to the /etc/profile of the server 

export http_proxy=http://1.2.3.4:8080
export https_proxy=http://1.2.3.4:8080
export no_proxy=$(echo $(cat /etc/hosts| grep -v ^# | grep -v ^$ | awk '{print $1}' | tr "\n" ",")localhost,127.0.0.1,.sujitnet11.net,.netx.sujit.com)

Please note the above one helps the undercloud and the other hosts' VIPs /IPs to be kept in the /etc/hosts of the KVM host, so that the subsequent access to the openstack CLI /GUI works from the KVM Host.

The line that was added to the /etc/yum.conf had been 

export proxy=http://1.2.3.4:8080

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Updating the CentOS 7 Operating System

Updating the CentOS 7 Operating System (Openstack tripleO KVM Host)

This is to update the CentOS7 Minimal install with the latest versions of the RPMs, before we can install the common utilities and the libvirt related RPMs. 

Post the server comes up we can proceed with the installation of libvirt, enabling libvirtd service. This will be followed by enabling the processor virtualization at the BIOS of the KVM Host and the nested virtualization enabling.

yum update -y && reboot

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Setting the hostname and the static IPs on the KVM host

Setting the hostname for the KVM Hypervisor.

Ensure that the hostname has been set using

hostnamectl set-hostname lab.example.com

Confirm the same using hostnamectl command, Also put an Entry of the IP address and the hostname in the /etc/hosts of the KVM Host.


Setting the Static IP on the KVM Host

Ensure that the IP address has been set for the KVM Host 

Set a static IP in the /etc/sysconfig/network-scripts/ifcfg-<DEVICE>

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Setting the SELINUX to permissive/enforcing mode on the KVM Host for the Pike tripleo Installation

Setting the SELINUX to permissive/enforcing mode on the KVM Host


SELINUX can be allowed to be in enforcing mode here in /etc/selinux/config

In this file set SELINUX=enforcing 

Followed by running the 

setenforce 1
sestatus 

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Enabling the processor virtualization feature on the KVM Host for OpenStack Pike TripleO installation

Enabling the processor virtualization feature on the KVM Host.

This is generally done at the BIOS of the Physical Server KVM host and once the same is enabled at the BIOS and the system is rebooted, the OS comes to know that. Please note that the enabling of the processor virtualization  feature menus in the BIOS will vary as per the vendor. 

Once the same is enabled and the server is rebooted, this cam be checked as follows.

Check the processor family on the server first

cat /proc/cpuinfo | grep -i vendor_id 

for me this shows GenuineIntel as my KVM host has the Intel CPUs. 

If there has been AMD CPUs, it would have shown Authentic AMD 

vendor_id       : AuthenticAMD


Depending on the above information, now you can check the CPUs for the enabled flags as you have already set the processor Virtualization in the BIOS of the KVM Host.

cat /proc/cpuinfo | grep -i -e svm -e vmx 

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts


As I have intel processors and the CPU Virtualization has been enabled in the BIOS, I have the processor flags showing 'vmx'. This signifies that the processor Virtualization has been successfully picked up by the OS of the KVM host.

Please note that if the KVM host had been having the CPUs from AMD, the enabled processor virtualization flag would have been 'svm' instead of 'vmx' above.

The enablement of the processor virtualization can also be seen using the virsh capabilities if you have the libvirt RPMs installed 


virsh capabilities | grep vmx


      <feature name='vmx'/>

Ensuring the needed processor Virtualization has been enabled for the KVM Host

Once the processor virtualizaton is enabled at the BIOS level and you reboot the KVM Host, the same can be verified using 

cat /proc/cpuinfo | grep -i -e svm -e vmx

As noted earlier, the 'svm' flag is for the processor virtualization enablement on the AMD processors and 'vmx' is for the processor virtualization feature enablement on the Intel processor. 

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Installation of the common utilities and the Libvirt RPMs on the KVM Host for OpenStack Pike TripleO installation

Installation of the common utilities and the Libvirt RPMs on the KVM host

As this is a minimal install of CentOS7, we will be installing few handy utilities and the libvirtd 

yum -y install vim bash-completion net-tools bind-utils curl wget rsync  libvirt qemu-kvm libvirt-client virt-install virt-manager virt-top virt-viewer


Enable and start the libvirtd daemon 

systemctl enable libvirtd
systemctl start libvirtd

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Enabling the Nested Virtualization on the KVM Host for OpenStack Pike Tripleo Installation

Enabling the Nested Virtualization on the KVM Host

This is needed on the KVM Host  as the nova instances that will be launched in the overcloud will be running on the overcloud nova nodes, that are themselves the virtual machines. 

So eventually it is going to be virtual machines running inside the virtual machines. So we are configuring the KVM host for the nested virtualization.

To have this we put the below entries in the file kvm-nested.conf in /etc/modprobe.d/ 

the file /etc/modprobe.d/kvm-nested.conf reads like 

options kvm_intel nested = 1

This is as I am having the Intel processors on the KVM Host.

In case there had been AMD processors, the same entry would have read as 

options kvm_amd nested = 1

Reboot the server once here.



Whether the KVM nested virtualization has been enabled or not.

on the KVM host 

cat /sys/module/kvm_intel/parameters/nested 

If the nested virtualization has been enabled you see a 'Y' here.


Alternative way to test the nested virtualization will be 


Using the virt-manager, create a VM and select option for CPU to 'Copy Host Configuration' followed by installation on CentOS minimal on the same.


Post install of the OS, login to the VM and if you can see the same CPU parameters as the Host, then you can be pretty sure that the nested virtualization is enabled. 

In the guest if cat /proc/cpuinfo | grep -i -e svm -e vmx shows the flags as svm or vmx then the nested virtualization has been enabled. This means that you can run VMs inside the guest.

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Creation of the Virtual network on the KVM host to which all the OpenStack Pike Overcloud and Undercloud VirtualMachines will connect to

Creation of the Virtual network on the KVM host 

On The KVM host, as libvirt has been installed and the libvirtd starts up, a default nework is seen which  in NAT and DHCP mode and is connected to a bridge 'virbro' to which the Physical interface of the KVM host is also connected.


Here we delete the 'default' network that has an IP of 192.168.122.1/24 and recreate that with a Class B IP address 172.16.0.1/16 belonging to the Network 172.16.0.0/16

Delete the existing 'default' interface.

virsh net-destroy default
virsh net-undefine default 

Create an XML file say nat.xml with the contents as 


<network>
  <forward dev='eno49' mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
    <interface dev='eno49'/>
  </forward>
  <bridge name='virbr2' stp='on' delay='0'/>
  <domain name='nat'/>
  <ip address='172.16.0.1' netmask='255.255.0.0'>
  </ip>
</network>

Save this file as say nat.xml 

The DHCP is disabled here as we want the undercloud server to be working as the DHCP for the overcloud servers during the introspection and the actual deployment.


The 'eno49' is the Physical interface on the KVM host that has the internet connectivity and this is connected to the Virtual bridge 'virbr2' on the KVM Host.

Define the new network using the XML file above 

virsh net-define nat.xml 

Confirm that the network is defined using 

virsh net-list 

Set the network to autostart and start the network-environment

virsh net-autostart nat 
virsh net-start nat


To see the network created 

virsh net-dumpxml nat 

<network connections='10'>
  <name>nat</name>
  <uuid>04640e45-0095-4877-a9bc-31970f8aa9d6</uuid>
  <forward dev='eno49' mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
    <interface dev='eno49'/>
  </forward>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:0a:89:16'/>
  <domain name='nat'/>
  <ip address='172.16.0.1' netmask='255.255.0.0'>
  </ip>
</network>


Confirm that you see a 'virbrX' interface here 'virbr0' with an IP address as in the nat.xml file that helped define the virtual network.

ip a s virbr2

ip a s virbr2
650: virbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 52:54:00:0a:89:16 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/16 brd 172.16.255.255 scope global virbr2

The virbr2 interface has IP of 172.16.0.1 and this is going to be used as the GW for the overcloud servers and all other VMs and as the DNS nameserver for all of them.

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Configure the OS installation media (CentOS 7 Minimal) on the KVM Host to Install the Pike TripleO undercloud node

Configure the OS installation media (CentOS 7 Minimal) on the KVM Host

Setting the CentOS Minimal media as the Virtual Media for OS installation on the undercloud VirtualMachine. This is the Virtual Media that will be used with the --location option of the virt-install command to install OS on the undercloud server.

Download the CentOS Minimal ISO image from the www.centos.org

Copy the ISO image to the /var/lib/libvirt/images to the KVM Host

ls -ltr /var/lib/libvirt/images/ | grep -i centos | grep -i min


-rw-r--r--. 1 qemu qemu   713031680 Sep  5  2017 CentOS-7-x86_64-Minimal-1611.iso

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Creation and OS Installation of the VirtualMachine that will work as the Undercloud server (OpenStack Pike TripleO)

Creation and OS Installation of the VirtualMachine that will work as the Undercloud server.

you can use command line of virsh or also the virt-manager GUI to create a Virtual Machine. 

I have already the CentOS7 Minimal Install media stored and known for the libvirt at /var/lib/libvirt/images on the KVM Host

For the undercloud, The VM created has 8 CPUs, 46GB RAM and 40 GB IDE HDD.

If you need the CLI here is the same.

virt-install --name pike1.sujitnet11.net --noautoconsole --virt-type qemu --memory 47102 --arch x86_64 --vcpus 8 --cpu host --location /var/lib/libvirt/images/CentOS-7-x86_64-Minimal-1611.iso --disk size=40,path=/var/lib/libvirt/images/pike1.sujitnet11.net,format=qcow2 --network network=nat

This installs the Virtual Machine with 46GB RAM, 8 CPUs, Mapping the CPU Host configuration to the VirtualMachine, with QEMU as the virtualization type. This uses the media (--location) /var/lib/libvirt/images/CentOS-7-x86_64-Minimal-1611.iso as the installation source. A Virtual disk of format qcow2, size 40GB gets created by this at the location /var/lib/libvirt/images/pike1.sujitnet11.net. Attachment of the Virtual Machine to the 'nat' network is done during this. The '--noautoconsole' says not to automatically connect to the console of the VirtualMachine after creation

root@win2k12r2 images]# virt-install  --name pike1.sujitnet11.net --noautoconsole --virt-type qemu --memory 47102 --arch x86_64 --vcpus 8 --cpu host --location /var/lib/libvirt/images/CentOS-7-x86_64-Minimal-1611.iso --disk size=40,path=/var/lib/libvirt/images/pike1.sujitnet11.net,format=qcow2 --network network=nat

Starting install...

Retrieving file .treeinfo...                                                                                    |  366 B  00:00:00

Retrieving file vmlinuz...                                                                                      | 5.1 MB  00:00:00

Retrieving file initrd.img...                                                                                   |  41 MB  00:00:00

Allocating 'pike1.sujitnet11.net'                                                                               |  40 GB  00:00:00

Domain installation still in progress. You can reconnect to

the console to complete the installation process.

[root@win2k12r2 images]#

Connect to the console for the installation part , I used the virt-manager to do so.

The Virtual Machinre reboots after the installation.

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Basic OS Configuration of the OpenStack Pike TripleO undercloud server before preparing it for the undercloud installation

These are the steps to configure basic OS things on the undercloud server before we prepare this for the undercloud installation.

Set the hostname of the Undercloud server (if not done during the OS installation)

hostnamectl set-hostname pike.sujitnet11.net 

Confirm the hostname has been set

hostnamectl 

Configuring the Static IP information on the undercloud server (if not done during the installation of the undercloud OS)


Ensure that the ethernet network interface of the server has a static configuration 

Edit the file /etc/sysconfig/network-scripts/ens3 to look like this 

NAME=ens3
DEVICE=ens3
BOOTPROTO=static
IPADDR=172.16.127.188
NETMASK=255.255.0.0
GATEWAY=172.16.0.1
DNS1=172.16.0.1

Disable the NetworkManager and Firewalld services and enable the network service on the undercloud

systemctl stop firewalld NetworkManager 
systemctl disabme firewalld NetworkManager 

Start and enable the network services 

systemctl enable network 
systemctl start network


Setting SELinux to permissive mode on the undercloud server

set /etc/selinux/config to be in permissive mode (id this is in disabled or enforcing mode earlier)

SELINUX=permissive 


set the SELINUX mode to 1 

setenforce 1

You should ideally reboot the server once as the SELINUX file context will be applied to the files at the system reboot.


Setting up the http_proxy and the https_proxy on the undercloud server 

This is needed as the undercloud server has to go to a proxy first before tryimg to reach to the internet.


Set the env proxies 

echo 'export http_proxy=http://1.2.3.4:8080
export https_proxy=http://1.2.3.4:8080
export no_proxy=$(echo $(cat /etc/hosts| grep -v ^# | grep -v ^$ | awk '{print $1}' | tr "\n" ",")localhost,127.0.0.1,.sujitnet11.net,.netx.sujit.com)' >> /etc/profile

echo 'export proxy=http://1.2.3.4:8080' >> /etc/yum.conf 

Logoff and log back in to activate the profile or also can do 'source /etc/profile'


OS update on the undercloud server and reboot 


Update the OS 

yum update -y 

Reboot the system 

systemctl reboot 

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)

Prepare the OpenStack Pike undercloud server for the undercloud installation

Prepare the OpenStack Pike undercloud server for the undercloud installation

These steps are done on the undercloud server.

Create the stack user for the undercloud installation 

useradd stack 

Set the password less SUDO for all the cokmad,
echo "stack ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/stack 
chmod 400 /etc/sudoers.d/stack 

--

Logon as the stack user to the undercloud server and create the SSH Keys

sudo su - stack
ssh-keygen -t rsa 

--
Copy the SSH keys for password less authentication to the KVM Libvirt Host (here our host is lab.example.com)


ssh-copy-id stack@lab.example.com

ssh-copy-id root@lab.example.com

This is needed for example if you want to remotely run any virsh commands remotely on the KVM host from the undercloud server, though this is not mandatory.
--



Install the repositories to be on the undercloud for the tripleo deployments 

Install the tripleo repos on the undercloud server 

sudo yum install -y https://trunk.rdoproject.org/centos7/current/python2-tripleo-repos-<version>.el7.centos.noarch.rpm

sudo yum -y install https://trunk.rdoproject.org/centos7/current/python2-tripleo-repos-0.0.1-0.20180319163358.8961edc.el7.centos.noarch.rpm

sudo rm -rf /var/cache/yum && yum repolist

--

Enabling the Tripleo repos on the undercloud server:

This is so as to have to enable the proper repos as per the release of the openstack on the undercloud server

As the stack user run the below (to be noted that we are going to install stable Pike OpenStack version on the undercloud)

sudo -E tripleo-repos -b pike current ceph


--

installation of the python-tripleoclient packages on the undercloud server

sudo yum -y install python-tripleoclient

This is good idea to do a yum cache clean up here

yum clean all 
rm -rf /var/cache/yum

If you want to use ceph Ansible to install and configure Ceph storage on the overcloud, then the same has to be installed and to be used for the Ceph depoyment in the undercloud

yum -y install ceph-ansible 

Go to the Main page of the OpenStack Tripleo on KVM : Openstack Pike TripleO undercloud deployment with overcloud controller HA on KVM (libvirt)