创建一个本地Kubernetes集群

关闭swap

sudo vim /etc/fstab 把带有 /swap.img 的一行注释掉, 然后重启 sudo reboot

不想重启的话可以适用这条命令关闭swap sudo swapoff -a

准备容器运行环境

# 安装Containerd
curl -vL https://github.com/containerd/containerd/releases/download/v1.6.33/containerd-1.6.33-linux-amd64.tar.gz -O
sudo tar -xzvf containerd-1.6.33-linux-amd64.tar.gz -C /usr/local
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo mv containerd.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now containerd

# 安装 runc
curl -vL https://github.com/opencontainers/runc/releases/download/v1.1.13/runc.amd64 -O
sudo install -m755 runc.amd64 /usr/local/sbin/runc

# 安装 CNI
sudo mkdir -p /opt/cni/bin
curl -vL https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-amd64-v1.5.1.tgz -O
sudo tar -xzvf cni-plugins-linux-amd64-v1.5.1.tgz -C /opt/cni/bin/

在root下执行 sudo su

1 2	mkdir /etc/containerd/ containerd config default > /etc/containerd/config.toml

编辑配置文件 vim /etc/containerd/config.toml

修改

...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false # <--- 这里的false改为true
...

保存后重启服务 sudo systemctl restart containerd

配置网络 sudo vim /etc/sysctl.d/k8s.conf

1	net.ipv4.ip_forward=1

使其生效: sudo sysctl --system

安装kubeadm


sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

启动kubelet

1	sudo systemctl enable --now kubelet

kubeadm创建集群 (可能需要一段时间)

这里的 --pod-network-cidr需要跟下面的flannel配置一样, flannel默认是 10.244.0.0/16

1	sudo kubeadm init --pod-network-cidr=10.77.0.0/16

看到这个就是创建成功了

1
2
3

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

如果需要将其他节点加入到集群内但是忘记复制了安装之后的命令, 可以创建一个新的 bootstrap token, 有效期为1天. 会打印出用于加入集群的命令.

1	kubeadm token create --print-join-command

复制 kubectl 用到的配置

1
2
3

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装 Flannel

1
2
3

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
sed -i s_10.244.0.0/16_10.77.0.0/16_g kube-flannel.yml
kubectl apply -f kube-flannel.yml

查看状态 kubectl get nodes

1 2	NAME STATUS ROLES AGE VERSION lsp-k8s-1 Ready control-plane 5m17s v1.30.3

注意 Flannel 在新版Ubuntu Server (24.04)上可能会安装失败, log报错如下:

1	Failed to check br_netfilter: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory

参考以下方案解决:

1	modprobe br_netfilter

安装 Istio (可选)

curl -L https://istio.io/downloadIstio | sh -

istioctl x precheck
istioctl install

整合脚本

整合脚本是我在写完前面内容之后一段时间才整理出来的, 因为一直忙活GKE所以忘了整本地的集群. 这里面的版本可能跟前文的不太一样.

这个脚本需要在所有节点上运行. 创建集群, 加入集群命令不一样就不放到这里了.

#!/bin/bash
set -euxo pipefail
sudo swapoff -a

curl -vL https://github.com/containerd/containerd/releases/download/v1.7.24/containerd-1.7.24-linux-amd64.tar.gz -O
sudo tar -xzvf containerd-1.7.24-linux-amd64.tar.gz -C /usr/local
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo mv containerd.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now containerd

curl -vL https://github.com/opencontainers/runc/releases/download/v1.2.2/runc.amd64 -O
sudo install -m755 runc.amd64 /usr/local/sbin/runc

sudo mkdir -p /opt/cni/bin
curl -vL https://github.com/containernetworking/plugins/releases/download/v1.6.0/cni-plugins-linux-amd64-v1.6.0.tgz -O
sudo tar -xzvf cni-plugins-linux-amd64-v1.6.0.tgz -C /opt/cni/bin/

sudo mkdir -p /etc/containerd/
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i s/SystemdCgroup\ =\ false/SystemdCgroup\ =\ true/g /etc/containerd/config.toml

sudo systemctl restart containerd

echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/k8s.conf
sudo sysctl --system

sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo systemctl enable --now kubelet

注意, PVE环境下如果图省事用Clone VM创建更多节点, 需要使用这样一个脚本来确保Clone出来的机器能够正常使用. 需要在上面的脚本之前执行并重启. 参考 Proxmox如何完整的复制一个VM

set -xe
sudo vim /etc/hostname
sudo vim /etc/hosts
echo -n | sudo tee /etc/machine-id
sudo rm /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
sudo rm -rf /etc/ssh/ssh_host*
sudo dpkg-reconfigure openssh-server

参考

Creating a cluster with kubeadm

Fix sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables.