本文更新于 2022-12-26，Kuberbetes 更新非常快，请注意可能有过时信息

前言

最近公司业务场景要一个高可用的集群节点用于工业环境，而且数据量非常大，刚刚好手上有一台 IDC 淘来的机架服务器，借机复习一下 K8s 的手动部署。

系统资源

虚拟机配置

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       901Mi        25Gi       4.0Mi       5.4Gi        30Gi
Swap:             0B          0B          0B
$ cat /proc/cpuinfo | grep -c processor
32

集群的安装

由于只安装一个 Control Panel，我们只需要配置好以下几个组件即可

  • ETCD
  • Kube-ApiServer
  • Kube-Controller-Manager
  • Kube-Scheduler
  • Kube-Proxy
  • Kubelet

基础的配置

安装基础软件

$ apt install bash-completion git net-tools sudo build-essential golang conntrack
$ go version # 这里的 Go 版本需要大于 1.12
go version go1.18.1 linux/amd64

设置NTP时间

$ sudo apt install chrony
$ sudo systemctl status chrony
● chrony.service - chrony, an NTP client/server
   Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2021-06-27 03:07:41 AKDT; 55s ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
 Main PID: 2028 (chronyd)
    Tasks: 2 (limit: 4915)
   Memory: 1.6M
   CGroup: /system.slice/chrony.service
           ├─2028 /usr/sbin/chronyd -F -1
           └─2029 /usr/sbin/chronyd -F -1

Jun 27 03:07:41 sxueck systemd[1]: Starting chrony, an NTP client/server...
Jun 27 03:07:41 sxueck chronyd[2028]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASJun 27 03:07:41 sxueck chronyd[2028]: Initial frequency 6.947 ppm
Jun 27 03:07:41 sxueck chronyd[2028]: Loaded seccomp filter
Jun 27 03:07:41 sxueck systemd[1]: Started chrony, an NTP client/server.
Jun 27 03:07:48 sxueck chronyd[2028]: Selected source 183.177.72.201
Jun 27 03:07:49 sxueck chronyd[2028]: Selected source 60.248.114.17
$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^- tock.ntp.infomaniak.ch        1  10   337   841   +491us[ +491us] +/-  110ms
^- time.cloudflare.com           3  10   377    72  -5089us[-5089us] +/-  121ms
^* 139.199.215.251               2  10   377  1017  -2594us[-3319us] +/-   40ms
^- makaki.miuku.net              2  10   333   24m    +78ms[  +77ms] +/-  106ms
$ sudo timedatectl set-timezone Asia/Shanghai

配置 L2 网桥给 CNI 插件使用

$ sudo /sbin/modprobe br_netfilter
$ cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
$ sudo sysctl -p /etc/sysctl.d/k8s.conf

至此，我们可以开始安装集群依赖组件

证书的生成

CA 根证书

K8s 的许多组件都需要依赖证书进行操作，例如 Kube-ApiServer 的认证等，我们这里通过一个自签 CA 证书进行整个集群的签名，但是请注意千万保管好这个证书

在 Kubernetes 集群中，各个组件间通过 TLS 进行通信。组件的证书可以代表其唯一性（ Common Name ）。搭建 PKI 可以自由签发证书给需要的组件。

安装 cfssl 证书生成工具

$ sudo mkdir /etc/kubernetes/pki -p
$ go env -w GO111MODULE=on
$ go env -w GOPROXY=https://goproxy.cn,direct # 可选步骤
$ go install github.com/cloudflare/cfssl/cmd/cfssl@latest
$ go install github.com/cloudflare/cfssl/cmd/cfssljson@latest
$ cp ~/go/bin/cfssl ~/go/bin/cfssljson /usr/local/bin
$ cfssl version
Version: dev
Runtime: go1.18.1

开始签署 CA 证书，为了图省事，反正都是自签证书，直接配置 Expiry 为 99 年

$ mkdir ~/ssl && cd ~/ssl

# 证书配置文件
$ cat > ca-ssl.json << EOF
{
    "signing": {
        "default": {
            "expiry": "867240h"
        },
        "profiles": {
            "kubernetes": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "867240h"
            }
        }
    }
}
EOF

# CA 签名申请
$ cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Guangdong",
            "L": "ShenZhen",
            "O": "k8s",
            "OU": "system"
        }
    ],
    "ca": {
        "expiry": "867240h"
    }
}
EOF

$ cfssl gencert -initca ca-csr.json| cfssljson -bare ca
2022/08/02 22:41:09 [INFO] generating a new CA key and certificate from CSR
2022/08/02 22:41:09 [INFO] generate received request
2022/08/02 22:41:09 [INFO] received CSR
2022/08/02 22:41:09 [INFO] generating key: rsa-2048
2022/08/02 22:41:10 [INFO] encoded CSR
2022/08/02 22:41:10 [INFO] signed certificate with serial number 224027550201873771575880785170986119805992526820

$ ll ca*pem
-rw------- 1 root root 1.7K Aug  2 22:41 ca-key.pem
-rw-r--r-- 1 root root 1.3K Aug  2 22:41 ca.pem

$ sudo cp ca*pem /etc/kubernetes/pki

Kube-ApiServer 认证证书

Kube-ApiServer 组件的证书一般都会作为集群的入口，例如使用 Kubectl 的时候就是与该证书进行签名匹配，如果出现不在签名范围的请求或者匹配异常则会出现 x509 错误

下面的签名请求配置中，hosts 字段的 IP 为所有 ApiServer 节点地址，这里可以预留几个 IP，以备以后扩容

$ cat > kubernetes-csr.json << EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "Kubernetes",
      "OU": "Kubernetes From Scratch",
      "ST": "BeiJing"
    }
  ]
}
EOF

$ cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-ssl.json \
  -hostname=${HOSTNAME},121.201.119.45,127.0.0.1,kubernetes.default \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes
2022/12/26 13:12:17 [INFO] generate received request
2022/12/26 13:12:17 [INFO] received CSR
2022/12/26 13:12:17 [INFO] generating key: rsa-2048
2022/12/26 13:12:18 [INFO] encoded CSR
2022/12/26 13:12:18 [INFO] signed certificate with serial number 262321052924478574973572968731187695389058393692

Kubelet 节点证书

该组件作为 Worker 节点的代理组件，运行在每个节点上，其作用定期从 Kube-ApiServer 组件接收新的或者修改的 Pod 规范，并确保 Pod 及其容器在规范标准下运行，因此有多少组件就要生成多少个 Kubelet 认证证书，同时节点通讯失败可以首先怀疑 Kubelet 问题。

这里使用脚本进行生成

$ NODES=$(cat <<-END
    master-1 127.0.0.1
END
)

function genpem() {
    hostname=$1
    ip=$2
    echo $hostname $ip

    cat > "kubelet/${hostname}-csr.json" <<EOF
    {
      "CN": "system:node:${hostname}",
      "key": {
        "algo": "rsa",
        "size": 2048
      },
      "names": [
        {
          "C": "CN",
          "L": "BeiJing",
          "O": "system:nodes",
          "OU": "Kubernetes From Scratch",
          "ST": "BeiJing"
        }
      ]
    }
EOF

    cfssl gencert \
      -ca=ca.pem \
      -ca-key=ca-key.pem \
      -config=ca-ssl.json \
      -hostname=${hostname},${ip} \
      -profile=kubernetes \
      ${hostname}-csr.json | cfssljson -bare ${hostname}
}

while read -r line; do
    a=($(echo "$line" | tr ' ' '\n'))
    hostname="${a[0]}"
    ip="${a[1]}"
    genpem $hostname $ip
done <<< "$NODES"

执行脚本即可生成

$ bash gen-kubelet.sh
master-1 127.0.0.1
2022/12/26 13:48:47 [INFO] generate received request
2022/12/26 13:48:47 [INFO] received CSR
2022/12/26 13:48:47 [INFO] generating key: rsa-2048
2022/12/26 13:48:47 [INFO] encoded CSR
2022/12/26 13:48:47 [INFO] signed certificate with serial number 161634764867939977984899022143501924846819499448

$ ls -al
...
-rw-r--r-- 1 idc idc 1110 Dec 26 13:48 master-1.csr
-rw------- 1 idc idc 1679 Dec 26 13:48 master-1-key.pem
-rw-rw-r-- 1 idc idc 1489 Dec 26 13:48 master-1.pem

其他证书配置

其他证书的配置大同小异，无非就是 Json 配置后用 cfssl 生产，这里一次性罗列出来

Kube-Controller-Manager 组件

$ cat > kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes From Scratch",
      "ST": "BeiJing"
    }
  ]
}
EOF

$ cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-ssl.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

Kube-Proxy 组件

$ cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "system:node-proxier",
      "OU": "Kubernetes From Scratch",
      "ST": "BeiJing"
    }
  ]
}
EOF

$ cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-ssl.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy