本文更新于 2024-12-26，Kubernetes 更新非常快，请注意可能有过时信息
前言
最近公司业务场景要一个高可用的集群节点用于工业环境，而且数据量非常大，刚刚好手上有一台 IDC 淘来的机架服务器，借机复习一下 K8s 的手动部署。
系统资源
虚拟机配置
$ free -h
total used free shared buff/cache available
Mem: 31Gi 901Mi 25Gi 4.0Mi 5.4Gi 30Gi
Swap: 0B 0B 0B
$ cat /proc/cpuinfo | grep -c processor
32
集群的安装
由于只安装一个 Control Panel，我们只需要配置好以下几个组件即可
- ETCD
- Kube-ApiServer
- Kube-Controller-Manager
- Kube-Scheduler
- Kube-Proxy
- Kubelet
基础的配置
安装基础软件
$ sudo apt install bash-completion git net-tools sudo build-essential golang conntrack
$ go version # 这里的 Go 版本需要大于 1.12
go version go1.22.2 linux/amd64
进行组件的编译
$ mkdir k8s && cd k8s
$ git clone https://github.com/kubernetes/kubernetes.git -b release-1.31 --depth 1
$ make all WHAT=cmd/kube-apiserver cmd/kube-controller-manager cmd/kube-scheduler cmd/cloud-controller-manager # 控制面编译，如果电脑性能足够可以直接不带参数编译全部
$ ls -al _output/bin/
total 979468
drwxr-xr-x 2 sxueck sxueck 4096 Oct 24 21:04 .
drwxr-xr-x 3 sxueck sxueck 4096 Oct 24 20:58 ..
-rwxr-xr-x 1 sxueck sxueck 69255320 Oct 24 21:04 apiextensions-apiserver
-rwxr-xr-x 1 sxueck sxueck 117022912 Oct 24 21:04 e2e.test
-rwxr-xr-x 1 sxueck sxueck 110219128 Oct 24 21:04 e2e_node.test
-rwxr-xr-x 1 sxueck sxueck 9351428 Oct 24 21:04 ginkgo
-rwxr-xr-x 1 sxueck sxueck 1896708 Oct 24 21:04 go-runner
-rwxr-xr-x 1 sxueck sxueck 66691224 Oct 24 21:04 kube-aggregator
-rwxr-xr-x 1 sxueck sxueck 90542232 Oct 24 21:04 kube-apiserver
-rwxr-xr-x 1 sxueck sxueck 84742296 Oct 24 21:04 kube-controller-manager
-rwxr-xr-x 1 sxueck sxueck 1683608 Oct 24 21:04 kube-log-runner
-rwxr-xr-x 1 sxueck sxueck 64417944 Oct 24 21:04 kube-proxy
-rwxr-xr-x 1 sxueck sxueck 63725720 Oct 24 21:04 kube-scheduler
-rwxr-xr-x 1 sxueck sxueck 58290328 Oct 24 21:04 kubeadm
-rwxr-xr-x 1 sxueck sxueck 56381592 Oct 24 21:04 kubectl
-rwxr-xr-x 1 sxueck sxueck 55070872 Oct 24 21:04 kubectl-convert
-rwxr-xr-x 1 sxueck sxueck 76902728 Oct 24 21:04 kubelet
-rwxr-xr-x 1 sxueck sxueck 75108504 Oct 24 21:04 kubemark
-rwxr-xr-x 1 sxueck sxueck 1601688 Oct 24 21:04 mounter
设置NTP时间
$ sudo apt install chrony
$ sudo systemctl status chrony
● chrony.service - chrony, an NTP client/server
Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-06-27 03:07:41 AKDT; 55s ago
Docs: man:chronyd(8)
man:chronyc(1)
man:chrony.conf(5)
Main PID: 2028 (chronyd)
Tasks: 2 (limit: 4915)
Memory: 1.6M
CGroup: /system.slice/chrony.service
├─2028 /usr/sbin/chronyd -F -1
└─2029 /usr/sbin/chronyd -F -1
Jun 27 03:07:41 sxueck systemd[1]: Starting chrony, an NTP client/server...
Jun 27 03:07:41 sxueck chronyd[2028]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASJun 27 03:07:41 sxueck chronyd[2028]: Initial frequency 6.947 ppm
Jun 27 03:07:41 sxueck chronyd[2028]: Loaded seccomp filter
Jun 27 03:07:41 sxueck systemd[1]: Started chrony, an NTP client/server.
Jun 27 03:07:48 sxueck chronyd[2028]: Selected source 183.177.72.201
Jun 27 03:07:49 sxueck chronyd[2028]: Selected source 60.248.114.17
$ chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- tock.ntp.infomaniak.ch 1 10 337 841 +491us[ +491us] +/- 110ms
^- time.cloudflare.com 3 10 377 72 -5089us[-5089us] +/- 121ms
^* 139.199.215.251 2 10 377 1017 -2594us[-3319us] +/- 40ms
^- makaki.miuku.net 2 10 333 24m +78ms[ +77ms] +/- 106ms
$ sudo timedatectl set-timezone Asia/Shanghai
修改一些必要的内核参数以供 CNI 插件能正常使用
$ sudo /sbin/modprobe br_netfilter
$ cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
$ sudo sysctl -p /etc/sysctl.d/k8s.conf
至此，我们可以开始安装集群依赖组件
证书的生成
CA 根证书
K8s 的许多组件都需要依赖证书进行操作，例如 Kube-ApiServer 的认证等，我们这里通过一个自签 CA 证书进行整个集群的签名，但是请注意千万保管好这个证书
在 Kubernetes 集群中，各个组件间通过 TLS 进行通信。组件的证书可以代表其唯一性（ Common Name ）。搭建 PKI 可以自由签发证书给需要的组件。
生成自签名的 CA 证书和私钥
$ sudo mkdir -p /etc/kubernetes/pki
$ sudo openssl genrsa -out /etc/kubernetes/pki/ca.key 2048
$ sudo openssl req -x509 -new -nodes -key /etc/kubernetes/pki/ca.key -subj "/CN=kubernetes-ca" -days 3650 -out /etc/kubernetes/pki/ca.crt
如果是以前的话，还需要使用 cf-ssl 工具，并且同时需要编写大量的证书 CSR，现在归功于大量的文档和 AI 搜索引擎，直接找到简易签发我们需要的证书方法了
生成其他组件的证书
kube-apiserver
$ sudo openssl genrsa -out /etc/kubernetes/pki/apiserver.key 2048
$ sudo openssl req -new -key /etc/kubernetes/pki/apiserver.key -subj "/CN=kube-apiserver" -out /etc/kubernetes/pki/apiserver.csr
$ sudo openssl x509 -req -in /etc/kubernetes/pki/apiserver.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out /etc/kubernetes/pki/apiserver.crt -days 365
Certificate request self-signature ok
subject=CN = kube-apiserver
kubelet
$ sudo openssl genrsa -out /etc/kubernetes/pki/kubelet-client.key 2048
$ sudo openssl req -new -key /etc/kubernetes/pki/kubelet-client.key -subj "/CN=kubelet-client" -out /etc/kubernetes/pki/kubelet-client.csr
$ sudo openssl x509 -req -in /etc/kubernetes/pki/kubelet-client.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out /etc/kubernetes/pki/kubelet-client.crt -days 365
Certificate request self-signature ok
subject=CN = kubelet-client
kube-controller-manager
$ sudo openssl genrsa -out /etc/kubernetes/pki/controller-manager.key 2048
$ sudo openssl req -new -key /etc/kubernetes/pki/controller-manager.key -subj "/CN=system:kube-controller-manager" -out /etc/kubernetes/pki/controller-manager.csr
$ sudo openssl x509 -req -in /etc/kubernetes/pki/controller-manager.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out /etc/kubernetes/pki/controller-manager.crt -days 365
Certificate request self-signature ok
subject=CN = system:kube-controller-manager
kube-scheduler
$ sudo openssl genrsa -out /etc/kubernetes/pki/scheduler.key 2048
$ sudo openssl req -new -key /etc/kubernetes/pki/scheduler.key -subj "/CN=system:kube-scheduler" -out /etc/kubernetes/pki/scheduler.csr
$ sudo openssl x509 -req -in /etc/kubernetes/pki/scheduler.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out /etc/kubernetes/pki/scheduler.crt -days 365
Certificate request self-signature ok
subject=CN = system:kube-scheduler
最后让我们来看一下具体签发和配置的证书文件
$ ls /etc/kubernetes/pki/ -al
total 68
drwxr-xr-x 2 root root 4096 Oct 24 21:36 .
drwxr-xr-x 3 root root 4096 Oct 24 21:20 ..
-rw-r--r-- 1 root root 1005 Oct 24 21:24 apiserver.crt
-rw-r--r-- 1 root root 899 Oct 24 21:24 apiserver.csr
-rw------- 1 root root 1704 Oct 24 21:23 apiserver.key
-rw-r--r-- 1 root root 1123 Oct 24 21:22 ca.crt
-rw------- 1 root root 1704 Oct 24 21:22 ca.key
-rw-r--r-- 1 root root 41 Oct 24 21:36 ca.srl
-rw-r--r-- 1 root root 1025 Oct 24 21:32 controller-manager.crt
-rw-r--r-- 1 root root 920 Oct 24 21:31 controller-manager.csr
-rw------- 1 root root 1704 Oct 24 21:31 controller-manager.key
-rw-r--r-- 1 root root 1005 Oct 24 21:31 kubelet-client.crt
-rw-r--r-- 1 root root 899 Oct 24 21:28 kubelet-client.csr
-rw------- 1 root root 1704 Oct 24 21:25 kubelet-client.key
-rw-r--r-- 1 root root 1013 Oct 24 21:36 scheduler.crt
-rw-r--r-- 1 root root 907 Oct 24 21:35 scheduler.csr
-rw------- 1 root root 1704 Oct 24 21:35 scheduler.key
组件的配置和启动
我们先将编译构建的二进制文件放到对应的目录中
$ sudo mkdir -p /etc/kubernetes/manifests /var/lib/kubelet /var/log/kubernetes
$ sudo cp _output/bin/kube-apiserver /usr/local/bin/
$ sudo cp _output/bin/kube-controller-manager /usr/local/bin/
$ sudo cp _output/bin/kube-scheduler /usr/local/bin/
$ sudo cp _output/bin/kubelet /usr/local/bin/
$ sudo cp _output/bin/kubectl /usr/local/bin/
在主节点上创建 /etc/systemd/system/kube-apiserver.service 文件
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--advertise-address=192.168.0.100 \ # 替换为你的主节点 IP
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--client-ca-file=/etc/kubernetes/pki/ca.crt \
--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
--etcd-servers=https://127.0.0.1:2379 \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
--requestheader-allowed-names=front-proxy-client \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--secure-port=6443 \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-cluster-ip-range=10.96.0.0/12 \
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
注意，--advertise-address 参数在 kube-apiserver 中非常重要，它指定了该组件向集群中的其他组件和客户端广播的 IP 地址。这个 IP 地址是其他组件（如 kubelet、kube-controller-manager、kube-scheduler 等）和外部客户端（如 kubectl）用来与 API Server 通信的地址，因此需要考虑一下选择一个和 Worker 节点共同的 IP 进行填写。
$ sudo systemctl daemon-reload
$ sudo systemctl start kube-apiserver
$ sudo systemctl enable kube-apiserver
$ sudo systemctl status kube-apiserver.service
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; preset: enabled)
Active: active (running) since Thu 2024-10-24 21:54:45 CST; 594us ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 68130 (9)
Tasks: 0 (limit: 28726)
Memory: 0B ()
CGroup: /system.slice/kube-apiserver.service
└─68130 /usr/lib/systemd/systemd-executor --deserialize 53 --log-level info --log-target journal
Oct 24 21:54:45 snival-pc systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 16.
Oct 24 21:54:45 snival-pc systemd[1]: Started kube-apiserver.service - Kubernetes API Server.