Kubernetes 1.10.0 multi-master installation With 3 Master and 3 Slaves Installation on CentOS7 with SSL - KVM Host and Guest preparations

KVM Host and Guest Preparations

KVM Host:

This part can be skipped if you are on another Virtualization Platform or bare metal. The idea is to have 3 Masters, 3 Slaves servers with CentOS installed and basic configuration done and ready to be installed Kubernetes.

KVM Host has CentOS minimal installation and on top it the libvirtd and the KDE Plasma Workspaces group.

NAT Network on the KVM Host.

The default network for NAT on the KVM Host was removed and a new NAT network was created.

virsh net-destroy default

virsh net-undefine default

Create an XML file with name natnetwork.xml (please note the KVM host has one interface connected only with the name eno49).

The NAT network being created here is 172.16.0.0/16 and this is going to be the network the Virtual Machines will connect to so as to have the Internet access.

<network connections='3'>

  <name>nat</name>

  <forward dev='eno49' mode='nat'>

    <nat>

      <port start='1024' end='65535'/>

    </nat>

    <interface dev='eno49'/>

  </forward>

  <bridge name='virbr2' stp='on' delay='0'/>

  <domain name='nat'/>

  <ip address='172.16.0.1' netmask='255.255.0.0'>

  </ip>

</network>

Create the NAT network on the KVM host/

virsh net-define -f natnetwork.xml

virsh net-autostart nat

virsh net-start nat

Create the CentOS7 Minimal Installation Media for installation of the Virtual Machines for Kubernetes.

Download the CentOS Minimal ISO image from the www.centos.org

Copy the ISO image to the /var/lib/libvirt/images to the KVM Host as CentOS-7-x86_64-Minimal-1611.iso.

ls -ltr /var/lib/libvirt/images/ | grep -i centos | grep -i min

Create the Virtual Machine that will work as the Base template

virt-install --name centos7.0 --noautoconsole --virt-type qemu --memory 16384 --arch x86_64 --vcpus 2 --cpu host --location /var/lib/libvirt/images/CentOS-7-x86_64-Minimal-1611.iso --disk size=20,path=/var/lib/libvirt/images/centos7.0.sujitnet11.net,format=qcow2 --network network=nat

Connect to virt-manager GUI to access the console and complete the installation.

Configuration of the VM to be working as the template

Once the Virtual Machine is up login to the Virtual Machine for the below configuration steps.

Set the Virtual Machine to have a static IP of 172.16.254.104/16 and default GW of 172.16.0.1 that is the NAT network interface on the host.

The following configuration of the Virtual Machine had been performed in addition to yum update -y to perform the full OS upgrade of the image.

There is a Virtual Machine image having CentOS7 minimal install with some needed Utilities installed such as vim, net-tools, bind-utils, iproute, yum-plugin-priorities and so on.

Other configurations on the image are:

·         The Chrony was setup and Configured to point to a correct time server

·         The SSH self-key for root user was generated and had been added to the local authorized keys

·         The KVM Hypervisor's SSH-Keys for the root had also been added to this VM

·         the /etc/profile had been set to have correct http_proxy, https_proxy and no_proxy

·         export http_proxy=http://myownproxy.mydomain.net:8080

·         export https_proxy=http://myownproxy.mydomain.net:8080

·         export no_proxy=$(echo $(cat /etc/hosts| grep -v ^# | grep -v ^$ | awk '{print $1}' | tr "\n" ",")localhost,127.0.0.1,.sujitnet11.net)

·         export no_proxy=$no_proxy,$( ip a s |grep -i scop | grep -i inet | awk '{print $2}' | awk -F '/' '{print $1}' | tr "\n" ",")localhost

·         the /etc/yum.conf was added

proxy=http://myownproxy.mydomain.net:8080

·         /etc/chromy.conf was edited to have the NTP server entry as

server 172.16.0.1 prefer iburst minpoll 4

Then chrony service was enabled.

This image is named as centos7.0 in the KVM hypervisor.

Post the configurations had been done the Virtual Machine had been shut down.

Using this image make the clones to get the Virtual Machines created for the masters and the slaves.

Creation of the KubeMaster virtual Machines (from the KVM Host)

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kubem1 -n kubem1

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kubem2 -n kubem2

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kubem3 -n kubem3

Creation of the slave (kubenetes nodes) Virtual Machines (from the KVM Host)

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kuben1 -n kuben1

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kuben2 -n kuben2

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kuben3 -n kuben3

Creation of the HAPROXY Virtual Machine on the KVM Host.

This setup is also going to use a HAPROXY based load balancer on a CentOS7 server. Using the same image to clone and get a VM for the haproxy server.

Here there is a single virtual machine working as HAPROXY load balancer thus is not highly available. You can refer to multiple references on the internet to see how to make HAPROXY as HA.

virt-clone -o centos7.0 --file /var/lib/libvirt/images/kube-haproxy -n kube-haproxy

As the Source image had an IP address of 172.16.254.204 and hostname of centos7.sujitnet11.net, this IP address and hostname has to be changed on each of the VMs.

Here all the VMs are started, the static IPs are changed and then they are rebooted one by one as shown below.

Configuration of the system hostname and static IP addresses for the Virtual Machines to be running Kubernetes

These all commands had been run from the KVM host where the Virtual Machines are hosted. Please note that the below commands had been run from the KVM Hypervisor host. The authorized keys of the KVM host had been added to the Virtual Machine being used as the source to clone these machines.

virsh start kubem1

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.221|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kubem1.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

sleep5

virsh start kubem2

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.222|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kubem2.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

sleep 5

virsh start kubem3

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.223|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kubem3.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

sleep 5

virsh start kuben1

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.224|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kuben1.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

sleep 5

virsh start kuben2

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.225|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kuben2.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

sleep 5

virsh start kuben3

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.226|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kuben3.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

virsh start kube-haproxy

sleep 20

ssh centos7 "sed -i -r -e 's|172.16.254.204|172.16.254.199|g' /etc/sysconfig/network-scripts/ifcfg-eth0"

ssh centos7 hostnamectl set-hostname kube-haproxy.sujitnet11.net

ssh centos7 cat /etc/sysconfig/network-scripts/ifcfg-eth0| grep IPADDR

ssh centos7 hostname

ssh centos7 reboot

Ensure that all the virtual machines are UP

From the KVM host

virsh list | grep -i kube

 842   kube-haproxy                   running

 860   kubem1                         running

 861   kubem2                         running

 862   kubem3                         running

 863   kuben1                         running

 864   kuben2                         running

 865   kuben3                         running

Configuring the Kernel Parameters for IP Forwarding on all the masters and slaves

for server in kubem{1..3} kuben{1..3}

do

ssh $server "echo 'net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1' >> /etc/sysctl.d/99-docker-sysctl.conf"

ssh $server "sysctl --system"

done

/etc/hosts on all the masters slaves and the kube-haproxy server. Please note that the IP 172.16.254.201 is the Kubernetes API server IP, which is an additional free IP on the servers’ subnet.

echo '172.16.254.221 kubem1.sujitnet11.net kubem1

172.16.254.222 kubem2.sujitnet11.net kubem2

172.16.254.223 kubem3.sujitnet11.net kubem3

172.16.254.224 kuben1.sujitnet11.net kuben1

172.16.254.225 kuben2.sujitnet11.net kuben2

172.16.254.226 kuben3.sujitnet11.net kuben3

172.16.254.199 kube-haproxy.sujitnet11.net kube-haproxy

172.16.254.201 k8s-api.virtual.local k8s-etcd.virtual.local k8s-api k8s-etcd' > /etc/hosts

On all the servers set the CentOS7 CBS Kubernetes Repository.

Please also note that here exclude to exclude the kubernetes 1.5 binaries.

echo '[virt7-container-common-candidate]

name=virt7-container-common-candidate

baseurl=http://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/

enabled=1

gpgcheck=0

# See CentOS-extras.repo - change that first, then make this match.

exclude=kubernetes*-1.5*' > /etc/yum.repos.d/kubernetes.repo

Install and configure Chrony for NTP synchronization on all the servers (if not done previously)

yum -y install chrony

systemctl enable chronyd

Please note: replace the IP mentioned here with the IP address of your NTP server. For me this IP is the IP of the NAT Virtual interface device Network on the KVM host and the same is the default gateway for all the Virtual Machines.

On all the machines.

echo 'server 172.16.0.1 prefer iburst minpoll 4' >> /etc/chrony.conf

systemctl restart chronyd

chronyc sources

Other relevant links for this documentation.

You can click on any of the link to view them.

The Main Document - Kubernetes 1.10.0 with 3 Master and Slave nodes and SSL on CentOS7

KVM Host and Guest Preprations

SSL Certificate Generations

Configure simple external HAPROXY

Configuring ETCD with SSL on the Master servers

Creation of the POD Network information in ETCD for flanneld

Install and Configure the Master Service on the Kubernetes Master servers

Installation and Configuration of the Kubernetes Slaves

Installation and testing of kube-dns

Configure the masters not to run kubernetes scheduled PODs