前言

最近公司业务场景要一个高可用的集群节点用于工业环境,而且数据量非常大,刚刚好手上有一台 IDC 淘来的机架服务器,借机复习一下 K8s 的手动部署。

系统资源

虚拟机配置

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       901Mi        25Gi       4.0Mi       5.4Gi        30Gi
Swap:             0B          0B          0B
$ cat /proc/cpuinfo | grep -c processor
32

集群的安装

由于只安装一个 Control Panel,我们只需要配置好以下几个组件即可

  • ETCD
  • Kube-ApiServer
  • Kube-Controller-Manager
  • Kube-Scheduler
  • Kube-Proxy
  • Kubelet

基础的配置

安装基础软件

$ apt install bash-completion git net-tools sudo build build-essential golang
$ go version # 这里的 Go 版本需要大于 1.12
go version go1.15.15 linux/amd64

设置NTP时间

$ sudo apt install chrony
$ sudo systemctl status chrony
● chrony.service - chrony, an NTP client/server
   Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2021-06-27 03:07:41 AKDT; 55s ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
 Main PID: 2028 (chronyd)
    Tasks: 2 (limit: 4915)
   Memory: 1.6M
   CGroup: /system.slice/chrony.service
           ├─2028 /usr/sbin/chronyd -F -1
           └─2029 /usr/sbin/chronyd -F -1

Jun 27 03:07:41 sxueck systemd[1]: Starting chrony, an NTP client/server...
Jun 27 03:07:41 sxueck chronyd[2028]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASJun 27 03:07:41 sxueck chronyd[2028]: Initial frequency 6.947 ppm
Jun 27 03:07:41 sxueck chronyd[2028]: Loaded seccomp filter
Jun 27 03:07:41 sxueck systemd[1]: Started chrony, an NTP client/server.
Jun 27 03:07:48 sxueck chronyd[2028]: Selected source 183.177.72.201
Jun 27 03:07:49 sxueck chronyd[2028]: Selected source 60.248.114.17
$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^- tock.ntp.infomaniak.ch        1  10   337   841   +491us[ +491us] +/-  110ms
^- time.cloudflare.com           3  10   377    72  -5089us[-5089us] +/-  121ms
^* 139.199.215.251               2  10   377  1017  -2594us[-3319us] +/-   40ms
^- makaki.miuku.net              2  10   333   24m    +78ms[  +77ms] +/-  106ms
$ sudo timedatectl set-timezone Asia/Shanghai

配置 L2 网桥给 CNI 插件使用

$ sudo /sbin/modprobe br_netfilter
$ cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
$ sudo sysctl -p /etc/sysctl.d/k8s.conf

至此,我们可以开始安装集群依赖组件

证书的生成

K8s 的许多组件都需要依赖证书进行操作,例如 Kube-ApiServer 的认证等,我们这里通过一个自签 CA 证书进行整个集群的签名,但是请注意千万保管好这个证书

安装 cfssl 证书生成工具

$ mkdir /etc/kubernetes/pki -p
$ go env -w GO111MODULE=on
$ go env -w GOPROXY=https://goproxy.cn,direct # 可选步骤
$ go get github.com/cloudflare/cfssl/cmd/cfssl
$ go get github.com/cloudflare/cfssl/cmd/cfssljson
$ cp ~/go/bin/cfssl ~/go/bin/cfssljson /usr/local/bin

开始签署 CA 证书,为了图省事,反正都是自签证书,直接配置 Expiry 为 99 年

$ mkdir ~/ssl && cd ~/ssl

# 证书配置文件
$ cat > ca-ssl.json << EOF
{
    "signing": {
        "default": {
            "expiry": "867240h"
        },
        "profiles": {
            "kubernetes": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "867240h"
            }
        }
    }
}
EOF

# CA 签名申请
$ cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Guangdong",
            "L": "ShenZhen",
            "O": "k8s",
            "OU": "system"
        }
    ],
    "ca": {
        "expiry": "867240h"
    }
}
EOF

$ cfssl gencert -initca ca-csr.json| cfssljson -bare ca
2022/08/02 22:41:09 [INFO] generating a new CA key and certificate from CSR
2022/08/02 22:41:09 [INFO] generate received request
2022/08/02 22:41:09 [INFO] received CSR
2022/08/02 22:41:09 [INFO] generating key: rsa-2048
2022/08/02 22:41:10 [INFO] encoded CSR
2022/08/02 22:41:10 [INFO] signed certificate with serial number 224027550201873771575880785170986119805992526820

$ ll ca*pem
-rw------- 1 root root 1.7K Aug  2 22:41 ca-key.pem
-rw-r--r-- 1 root root 1.3K Aug  2 22:41 ca.pem

$ cp ca*pem /etc/kubernetes/pki

Kube-ApiServer 认证证书

Kube-ApiServer 组件的证书一般都会作为集群的入口,例如使用 Kubectl 的时候就是与该证书进行签名匹配,如果出现不在签名范围的请求或者匹配异常则会出现 x509 错误

下面的签名请求配置中,hosts 字段的 IP 为所有 ApiServer 节点地址,这里可以预留几个 IP,以备以后扩容