Sunday, July 1, 2018

Kubernetes 1.10.0 multi-master installation with 3 Master and 3 Slaves installation on CentOS7 with SSL - SSL Certificates generation


SSL Certificate Generation
Certificates generation

Here we generate the SSL Certificates to be used by Kubernetes and its components. As we plan also to have ETCD to have SSL secured communications we also generate the SSL certificates to be used by ETCD.

Please note that the certificate generation is being done from the first master node only. Finally we copy over the certificates to the other nodes.

Downloading the CFSSL binaries for SSL certificate generation.
wget  https://pkg.cfssl.org/R1.2/cfssl_linux-amd64  https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
sudo mv cfssl_linux-amd64 /usr/bin/cfssl
sudo mv cfssljson_linux-amd64 /usr/bin/cfssljson

Creation of the Certificate Authority that will be used to sign the SSL Servers and the client certificates.
Certificate Authority
mkdir -p /srv/kubernetes
cd /srv/kubernetes

Generate the CA configuration file, certificate, and private key:

{
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

Create the CA certificate signing request.
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Karnataka"
    }
  ]
}
EOF


Generate the CA Certificate
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

Client and Server Certificates
Here the client and server certificates are generated. Also client certificate for the admin user is generated here.
cd /srv/kubernetes
Generate the admin client certificate and private key:
Signing request for the certificate
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "system:masters",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF

Signing the certificate with the CA certificate.
cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin

The Kubelet Client Certificates
Kubernetes uses a special-purpose authorization mode called Node Authorizer that specifically authorizes API requests made by Kubelets. In order to be authorized by the Node Authorizer, Kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName>. In this section you will create a certificate for each Kubernetes worker node that meets the Node Authorizer requirements.

Generate a certificate and private key for each Kubernetes worker node:

for instance in kubem{1..3}.sujitnet11.net kuben{1..3}.sujitnet11.net ; do
cat > ${instance}-csr.json <<EOF
{
  "CN": "system:node:${instance}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "system:nodes",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF
# Get the IP of the host from the /etc/hosts
# Get the hostname also from the /etc/hosts as per the name
INTERNAL_IP=$(cat /etc/hosts | grep $instance | awk '{print $1}')
NAME=$(cat /etc/hosts | grep $instance | awk '{print $2}' | cut -d . -f 1)

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${instance},${INTERNAL_IP},${NAME} \
  -profile=kubernetes \
  ${instance}-csr.json | cfssljson -bare ${instance}
done

The Controller Manager Client Certificate
Generate the kube-controller-manager client certificate and private key:

cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "system:kube-controller-manager",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

}


The Kube Proxy Client Certificate
Generate the kube-proxy client certificate and private key:

{
cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "system:node-proxier",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy

}

The Scheduler Client Certificate
Generate the kube-scheduler client certificate and private key:

{
cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "system:kube-scheduler",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler

}

The Kubernetes API Server Certificate
The k8s-api.sujitnet11.net static IP address 172.16.254.201 will be included in the list of subject alternative names for the Kubernetes API Server certificate. This will ensure the certificate can be validated by remote clients.
This is the IP address that will be available on the kube-haproxy server and is managed by the haproxy service running over there. This IP is where the kubernetes API server service will keep listening and haproxy will forward the requests to the kube-apiserver IP addresses of each of the master nodes.

Please note the --hostname in the cfssl gencert below. This contains the FQDNs and the IP addresses of the master nodes and that of the Kubernetes API server address that will reside on the haproxy server. The IP 100.0.65.1 is also included here. 100.65.0.1 is the IP address of the first service of Kubernetes and this is the first IP from the Kubernetes Service Subnet as per our plan 100.65.0.0/24.

Generate the Kubernetes API Server certificate and private key:

{

KUBERNETES_PUBLIC_ADDRESS=172.16.254.201

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "Kubernetes",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
 -hostname=100.65.0.1,172.16.254.221,172.16.254.222,172.16.254.223,kubem1.sujitnet11.net,kubem2.sujitnet11.net,kubem3.sujitnet11.net,${KUBERNETES_PUBLIC_ADDRESS},127.0.0.1,kubernetes.default \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes

}

The Service Account Key Pair
The Kubernetes Controller Manager leverages a key pair to generate and sign service account tokens as describe in the managing service accounts documentation.

Generate the service-account certificate and private key:

{

cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "IN",
      "L": "Bangalore",
      "O": "Kubernetes",
      "OU": "k8s-api.sujitnet11.net",
      "ST": "Karnataka"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

}


Copy over the certificates to all the servers
for U in kubem{1..3} kuben{1..3} kube-haproxy;  do ssh $U "scp -pr /srv/kubernetes $U:/root/; done





Other relevant links for this documentation.

You can click on any of the link to view them.

 The Main Document - Kubernetes 1.10.0 with 3 Master and Slave nodes and SSL on CentOS7

 KVM Host and Guest Preprations

 SSL Certificate Generations

 Configure simple external HAPROXY

 Configuring ETCD with SSL on the Master servers

 Creation of the POD Network information in ETCD for flanneld

 Install and Configure the Master Service on the Kubernetes Master servers

 Installation and Configuration of the Kubernetes Slaves

 Installation and testing of kube-dns

 Configure the masters not to run kubernetes scheduled PODs

Posted by at
Labels: , , , , , , , , , , ,

1 comment:

  1. The Skill PediaMarch 22, 2023 at 1:24 AM

    Thanks for sharing this useful information this information will be very helpful for us.
    Openstack Training

    ReplyDelete

Subscribe to: Post Comments (Atom)